Likes (4)

Comment

Share

I am using NixOs as a Development environment that is running on Google Cloud VM.
NixOs official Web site does not provide an official GCP image. On the NixOs Wiki page, you can find an article how you can build your GCP NixOs Image that didn’t work well in the past but now seems all build image issue was resolved and now works well. NixOS Wiki page article also provides two GCP storage that you can find outdated GCP VM images that seem not actively maintained anymore.

• nixos-images > gs://nixos-images
• nixos-cloud-images > gs://nixos-images

In the below "HowTo," we will cover how you can build your images in GCP by using Google Cloud VM and store the images in your Google storage bucket.

Preparing the Google Cloud Environment.

Before we can start, you will need to have a Google Cloud Account.

We will create a dedicated GCP Project with VM that we will use to build updated NixOs images from the latest nixpkgs builds.

Prerequisites

Before starting
In the 1st step, you will need to install Google Cloud SDK Depending on the OS or Linux Distribution, and please follow installation instructions.

Gcloud Authentication

Once installation is complete, you will need to authenticate CLI to be able to access your google cloud resources.

~> gcloud auth login
Go to the following link in your browser:
 https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32542940657.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=skHTaEGrhJSYIzDIwt4phrhSzUm97t&prompt=consent&access_type=offline&code_challenge=EcgLC0aZhpefFkL7k6ep-8lvJ1Og8NgCvs9VllOE5lQ&code_challenge_method=S256

Enter verification code: 

GCP Project Creation

Once authenticated, you should be able to access all cloud resources in your google cloud account.
If you are using your account for the first time, Google will create a default Project for your account. We will ignore the default project and will make a dedicated project to build NioxOs images.

GCP New Project Creation

We will create a dedicated GCP project. I use the name for the project mudrii-nixos.

~> gcloud projects create mudrii-nixos 
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/mudrii-nixos].
Waiting for [operations/cp.8533728690478532386] to finish...done. Enabling service [cloudapis.googleapis.com] on project [mudrii-nixos]...
Operation "operations/acf.p2-237810657129-fa1f0fa7-3015-46ff-ab52-f92d7bd3f8df" finished successfully.

Get details on the created project:

 ~> gcloud projects describe mudrii-nixos
createTime: '2021-09-06T08:57:16.001Z'
lifecycleState: ACTIVE
name: mudrii-nixos
projectId: mudrii-nixos
projectNumber: '111111111111'

~> gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
mudrii-nixos mudrii-nixos 111111111111

Configuring Gcloud CLI

Once a new project is created, it is a good practice to add the newly created project as default for Gcloud CLI. Every command executed with Gcloud of Gsutils will perform on the newly created project.

~> gcloud config set project mudrii-nixos
Updated property [core/project].

Configuring Project Billing

One more step is needed to make the GCP project usable; we will need to add billing to the project.

Listing the Available Billing Accounts

~> gcloud alpha billing accounts list
ACCOUNT_ID NAME OPEN MASTER_ACCOUNT_ID
ZZZZZ-ZZZZZZZ-ZZZZZZ MyBill True

Associating the Billing Account With the Project

~> gcloud alpha billing accounts projects link mudrii-nixos --billing-account=ZZZZZ-ZZZZZZZ-ZZZZZZ 
billingAccountName: billingAccounts/ZZZZZ-ZZZZZZZ-ZZZZZZ 
billingEnabled: true
name: projects/mudrii-nixos/billingInfo
projectId: mudrii-nixos

Verifying the Billing Account Association

~> gcloud alpha billing accounts projects list --billing-account=ZZZZZZ-ZZZZZZ-ZZZZZZ
PROJECT_ID BILLING_ACCOUNT_ID BILLING_ENABLED
mudrii-nixos ZZZZZ-ZZZZZZZ-ZZZZZZ True

Creating and Configuring Google Storage Bucket

The next step is to create a Google storage bucket to store generated nixos GCP VM images.

~> gsutil mb gs://nixos-images-gcp
gsutil mb gs://nixos-images-gcp


~> gsutil du -s -h gs://nixos-images-gcp

I decided to grant read access to external users to the generated images. If you do not intend to share generated images, you can skip the below step.

~> gsutil iam ch allUsers:objectViewer gs://nixos-images-gcp

Creating and Configuring GCP Networking

By default, Google will create a global VPC and subnets for every region and firewall rules once a project is created.

To make it clean, I decided to create a separate network stack with a subnet and firewall to control the resources better.

Before creating a network, a good practice to remove all automated default network VPC, subnets, and firewalls.

Listing the Available Network Resource

List VPC:

~> gcloud compute networks list
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
default AUTO REGIONAL

Listing Subnets

~> gcloud compute networks subnets list
NAME REGION NETWORK RANGE
default us-central1 default 10.128.0.0/20
default europe-west1 default 10.132.0.0/20
default us-west1 default 10.138.0.0/20
default asia-east1 default 10.140.0.0/20
default us-east1 default 10.142.0.0/20
default asia-northeast1 default 10.146.0.0/20
default asia-southeast1 default 10.148.0.0/20
default us-east4 default 10.150.0.0/20
default australia-southeast1 default 10.152.0.0/20
default europe-west2 default 10.154.0.0/20
default europe-west3 default 10.156.0.0/20
default southamerica-east1 default 10.158.0.0/20
default asia-south1 default 10.160.0.0/20
default northamerica-northeast1 default 10.162.0.0/20
default europe-west4 default 10.164.0.0/20
default europe-north1 default 10.166.0.0/20
default us-west2 default 10.168.0.0/20
default asia-east2 default 10.170.0.0/20
default europe-west6 default 10.172.0.0/20
default asia-northeast2 default 10.174.0.0/20
default asia-northeast3 default 10.178.0.0/20
default us-west3 default 10.180.0.0/20
default us-west4 default 10.182.0.0/20
default asia-southeast2 default 10.184.0.0/20
default europe-central2 default 10.186.0.0/20
default northamerica-northeast2 default 10.188.0.0/20
default asia-south2 default 10.190.0.0/20
default australia-southeast2 default 10.192.0.0/20

Listing Firewalls

~> gcloud compute firewall-rules list 
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
default-allow-icmp default INGRESS 65534 icmp False
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 tcp:3389 False
default-allow-ssh default INGRESS 65534 tcp:22 False

To show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.

Removing the Default Created Network Resource

Before removing VPC, we need to remove the firewall first, and once all firewall rules have been removed, we can remove the default VPC.

Removing the Default Firewall

~> gcloud compute firewall-rules delete default-allow-internal --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-internal].


~> gcloud compute firewall-rules delete default-allow-icmp --quiet 
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-icmp].


~> gcloud compute firewall-rules delete default-allow-rdp --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-rdp].


~> gcloud compute firewall-rules delete default-allow-ssh --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-ssh].

Removing Default VPC

~> gcloud compute networks delete default --quiet
Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/default].

Creating Networking Resources

We cleared all default networks stuck, and now we can create a new network stuck.

Creating a VPC:

Now, we are ready to create a custom VPC.

~> gcloud compute networks create vpc-nixos --subnet-mode=custom 
Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/vpc-nixos].
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
vpc-nixos CUSTOM REGIONAL

Instances on this network will not be reachable until firewall rules
are created. As an example, you can allow all internal traffic between
instances as well as SSH, RDP, and ICMP by running:

Creating a Custom Subnet

In the below example, I use a narrow IP range in Asia-southeast1 “Singapore” region. As I will run a single instance that will generate nixos images, I do not intend to run anything else.

~> gcloud compute networks subnets create subnet-sg --network=vpc-nixos --range=192.168.1.0/24 --region asia-southeast1
Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/regions/asia-southeast1/subnetworks/subnet-sg].
NAME REGION NETWORK RANGE
subnet-sg asia-southeast1 vpc-nixos 192.168.1.0/24

Creating a Firewall Rule to Allow SSH

In order to connect to remote VM over ssh, we need to open a firewall on port 22.

~> gcloud compute --project=mudrii-nixos firewall-rules create allow-ssh --direction=INGRESS --priority=1000 --network=vpc-nixos --action=ALLOW --rules=tcp:22 --target-tags=allow-ssh
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/allow-ssh].
Creating firewall...done.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
allow-ssh vpc-nixos INGRESS 1000 tcp:22 False

Adding an Image to Your VM Images Repository to be Able to Create NixOs VM in GCP

To create VM with NixOS installed, we need to have a VM image in our GCP images repositories.

In the below example, I am using one of the images I created previously locate in gs://nixos-images-gcp/

Note: make sure you specify your project ID --project=mudrii-nixos

~> gcloud compute images create nixos-21-05-2873 \
--source-uri=gs://nixos-images-gcp/nixos-image-21.05.2873.6bfe71f2a4e-x86_64-linux.raw.tar.gz \
--description=nixos-image-21.05.2873.6bfe71f2a4e \
--family=nixos \
--project=mudrii-nixos

List available images in your image repository.

 ~> gcloud compute images list | grep nixos
nixos-image-21-05-2873-6bfe71f2a4e-x86-64-linux mudrii-nixos nixos-image-21-05 READY

Creating NixOS VM

Our network stuck is up a ready we can create a dedicated VM to build nixos GCP images.

Here are a few points to mention:

• VM will be created in the same region where we set our subnet. --zone=asia-southeast1-b
• --machine-type=n2d-standard-4 VM AMD EMYC 4 vCPU and 16 GB RAM
• --metadata=enable-oslogin=true allow you to ssh into the VM with your Gcloud account
• --tags=allow-ssh Added tag to the VM to allow open port ssh we configured in the firewall
• --boot-disk-size=60GB size of the Disk you may select a lower capacity I find 30 works well too
• --boot-disk-type=pd-ssd This is the fastest option local SSD will speed up significantly image generation

~> gcloud beta compute --project=mudrii-nixos instances create nixos-base --zone=asia-southeast1-b --machine-type=n2d-standard-4 --subnet=subnet-sg --network-tier=PREMIUM --metadata=enable-oslogin=true --tags=allow-ssh --image=nixos-21-05-2873 --image-project=mudrii-nixos --boot-disk-size=60GB --boot-disk-type=pd-ssd --boot-disk-device-name=nixos-base
Created [https://www.googleapis.com/compute/beta/projects/mudrii-nixos/zones/asia-southeast1-b/instances/nixos-base].
WARNING: Some requests generated warnings:
 - Disk size: '60 GB' is larger than image size: '3 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
nixos-base asia-southeast1-b n2d-standard-4 192.168.1.2 35.188.151.168 RUNNING

SSH into Remote

We can ssh directly to the newly created VM with the gcloud command:

~> gcloud compute ssh --project=mudrii-nixos nixos-base

Building the NixOS Image

Once SSH gets into the newly created NixOs Google VM, we can start preparing to build the NixoS CGP image.

Authenticate from VM into Google Cloud

First, we need to become root.

~> sudo -i 

We need access to google SDK to upload newly created images into the google bucket created in the previous steps. To authenticate, we need google SDK, and instead of installing, we will use nix-shell.

[root@nixos-base:~]# nix-shell -p google-cloud-sdk git

Once we have access to the Gcloud binary, we can auth into the google cloud.

[nix-shell:~]# gcloud auth login


[nix-shell:~]# gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
mudrii-nixos mudrii-nixos 111111111111


[nix-shell:~]# gcloud config set project mudrii-nixos
Updated property [core/project].


[nix-shell:~]# gcloud config configurations list
NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION
default True [email protected] mudrii-nixos

To validate authentication, you can check if you have access to your Gcloud bucket created in the previous step.

Note: make sure you add BOTO_CONFIG=/dev/null or you will get an error during the image build.

[nix-shell:~]# export BOTO_CONFIG=/dev/null
[nix-shell:~]# gsutil ls -l gs://nixos-images-gcp

Update NixOS to the Latest Version

To create an image with the latest stable NixOs is essential to update the existing version with the latest version.

Check existing NixOs version and Linux Kernel.

[nix-shell:~]# nixos-version
21.05.3001.12eb1d16ae3 (Okapi)


[nix-shell:~]# uname -a
Linux nixos-base.asia-southeast1-b.c.mudrii-nixos.internal 5.10.62 #1-NixOS SMP Fri Sep 3 08:09:31 UTC 2021 x86_64 GNU/Linux

Updating NixOs to the Latest Version

Update 1st nix-channels:

[nix-shell:~]# nix-channel --list
nixos https://nixos.org/channels/nixos-21.05


[nix-shell:~]# nix-channel --update 
unpacking channels...
created 1 symlinks in user environment

Update the first NixOS:

[nix-shell:~]# nixos-rebuild switch
building Nix...
building the system configuration...
these derivations will be built:
...
..
.

[nix-shell:~]# nixos-version
21.05.3021.8b0b81dab17 (Okapi)

Or, if the version is outdated significantly, run the following:

[nix-shell:~]# nixos-rebuild switch --upgrade

NixOs Cleanup

Before attempting image creation, it is a good idea to do a cleanup and do a garbage collection on existing updates and repair any sha inconsistency that may cause failed image build.

[nix-shell:~]# nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/system
removing generation 4
removing old generations of profile /nix/var/nix/profiles/per-user/root/channels
removing generation 5
finding garbage collector roots...
removing stale link from 
...
..
.
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 21.60 MiB
167 store paths deleted, 1129.80 MiB freed




[nix-shell:~]# nix-store --gc
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/trash'
deleting unused links...
note: currently hard linking saves 21.60 MiB
0 store paths deleted, 0.00 MiB freed




[nix-shell:~]# nix-store --repair --verify --check-contents
reading the Nix store...
checking path existence...
checking hashes...
path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' was modified! expected hash 'sha256:0dvxzzklaswx0d2svx0nzjilqfmgd2dxffi7hcbz89p7r6w1jab0', got 'sha256:1wpsb25jajbqvsw29jy073w0cr994005xb22pb9p8mkinn9vx8hp'
copying path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' from 'https://cache.nixos.org'...




[nix-shell:~]# nix-store --optimise
430.37 MiB freed by hard-linking 47107 files

NixOs GCP Image Build

To build a GCP nixos image, we need to clone the nixpkgs repository where build scripts is located:

[nix-shell:~]# git clone https://github.com/NixOS/nixpkgs.git --depth 1
Cloning into 'nixpkgs'...
remote: Enumerating objects: 45763, done.
remote: Counting objects: 100% (45763/45763), done.
remote: Compressing objects: 100% (29848/29848), done.
remote: Total 45763 (delta 1593), reused 38844 (delta 1293), pack-reused 0
Receiving objects: 100% (45763/45763), 30.46 MiB | 14.19 MiB/s, done.
Resolving deltas: 100% (1593/1593), done.
Updating files: 100% (27800/27800), done.

The final step is to run a script that will build a GCP nixos image based on the underlying system version.

Note: Make sure you specify your GCP storage bucket name in BUCKET_NAME=

[nix-shell:~]# BUCKET_NAME=nixos-images-gcp nixpkgs/nixos/maintainers/scripts/gce/create-gce.sh
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
 /nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev
copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
...
..
.
/nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image
gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz

NixOs Image Build Verification

Once the image is created and uploaded into Google storage bucket and in the image repository, we can verify by the following:

[nix-shell:~]# gsutil ls -l gs://nixos-images-gcp
 434150003 2021-09-12T03:18:26Z gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz

Verify Image repository:

[nix-shell:~]# gcloud compute images list | grep nixos
nixos-image-21-05-3021-8b0b81dab17-x86-64-linux mudrii-nixos nixos-image-21-05 READY

Post-Build Cleanup

It is good to remove the nixpkgs repository to minimize space and speed on the next build and stop VM to reduce the cost on your cloud bill.

[nix-shell:~]# ls -la
total 44
drwx------ 8 root root 4096 Sep 13 11:02 .
drwxr-xr-x 17 root root 4096 Sep 6 10:55 ..
-rw------- 1 root root 2538 Sep 10 10:27 .bash_history
drwxr-xr-x 3 root root 4096 Sep 6 11:07 .cache
drwxr-xr-x 3 root root 4096 Sep 6 11:07 .config
lrwxrwxrwx 1 root root 64 Sep 13 11:02 gce -> /nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image
drwxr-xr-x 3 root root 4096 Sep 6 11:48 .gsutil
-rw-r--r-- 1 root root 45 Aug 25 05:05 .nix-channels
drwx------ 2 root root 4096 Sep 13 10:30 .nix-defexpr
drwxr-xr-x 9 root root 4096 Sep 13 10:55 nixpkgs
drwx------ 2 root root 4096 Sep 6 10:55 .ssh


[nix-shell:~]# rm -rf ~/nixpkgs


[nix-shell:~]# sudo shutdown now
Connection to 35.198.250.170 closed by remote host.
Connection to 35.198.250.170 closed.
ERROR: (gcloud.compute.ssh) [/run/current-system/sw/bin/ssh] exited with return code [255].

Fin

Next time, when you want to create a new VM on Google Cloud, you can specify in custom images your latest nixos image. Make sure you add --metadata=enable-oslogin=true to allow you to ssh into the VM.