Airship Treasuremap¶

Airship is a collection of components that coordinate to form means of configuring and deploying and maintaining a Kubernetes environment using a declarative set of yaml documents.

More specifically, the current focus of this project is the implementation of OpenStack on Kubernetes (OOK).

Component Projects¶

Pegleg¶

Pegleg is a document aggregator that provides early linting and validations via Deckhand, a document management micro-service within Airship.

Shipyard¶

Shipyard is the directed acyclic graph controller for Kubernetes and OpenStack control plane life cycle management.

Shipyard provides the entrypoint for the following aspects of the control plane:

Designs and Secrets¶

Site designs, including the configuration of bare metal host nodes, network design, operating systems, Kubernetes nodes, Armada manifests, Helm charts, and any other descriptors that define the build out of a group of servers enter the Airship via Shipyard. Secrets, such as passwords and certificates, use the same mechanism. The designs and secrets are stored in Airship’s Deckhand, providing for version history and secure storage among other document-based conveniences.

Actions¶

Interaction with the site’s control plane is done via invocation of actions in Shipyard. Each action is backed by a workflow implemented as a directed acyclic graph (DAG) that runs using Apache Airflow. Shipyard provides a mechanism to monitor and control the execution of the workflow.

Drydock¶

Drydock is a provisioning orchestrator for baremetal servers that translates a YAML-based declaritive site topology into a physical undercloud that can be used for building out a enterprise Kubernetes cluster. It uses plugins to leverage existing provisioning systems to build the servers allowing integration with the provisioning system that best fits the goals and environment of a site.

Capabilities¶

Initial IPMI configuration for PXE booting new servers.
Support for Canonical MAAS provisioning.
Configuration of complex network topologies including bonding, tagged VLANs and static routes
Support for running behind a corporate proxy
Extensible boot action system for placing files and SystemD units on nodes for post-deployment execution
Supports Keystone-based authentication and authorization

Deckhand¶

Deckhand is a document-based configuration storage service built with auditability and validation in mind.

Core Responsibilities¶

layering - helps reduce duplication in configuration by applying the notion of inheritance to documents
substitution - provides separation between secret data and other configuration data for security purposes and reduces data duplication by allowing common data to be defined once and substituted elsewhere dynamically
revision history - maintains well-defined collections of documents within immutable revisions that are meant to operate together, while providing the ability to rollback to previous revisions
validation - allows services to implement and register different kinds of validations and report errors
secret management - leverages existing OpenStack APIs – namely Barbican – to reliably and securely store sensitive data

Armada¶

Armada is a tool for managing multiple Helm charts with dependencies by centralizing all configurations in a single Armada YAML and providing life-cycle hooks for all Helm releases.

Core Responsibilities¶

Multiple Chart Deployments and Upgrades driven by Armada Manifests
Manage multiple chart dependencies using Chart Groups
Enhancing base Helm functionality
Supports Keystone-based authentication and authorization

Kubernetes¶

Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.

Promenade¶

Promenade is a tool for bootstrapping a resilient, self-hosted Kubernetes cluster and managing its life-cycle.

Bootstrapping begins by provisioning a single-node cluster with a complete, configurable Airship infrastructure. After hosts are added to the cluster, the original bootstrapping node can be re-provisioned to avoid subtle differences that could result in future issues.

Promenade provides cluster resiliency against both node failures and full cluster restarts. It does so by leveraging Helm charts to manage core Kubernetes assets directly on each host, to ensure their availability.

Helm¶

Helm is a package manager for Kubernetes. It helps you define, install, and upgrade even the most complex Kubernetes applications using Helm charts.

A chart is a collection of files that describe a related set of Kubernetes resources. Helm wraps up each chart’s deployment into a concrete release, a tidy little box that is a collection of all the Kubernetes resources that compose that service, and so you can interact with a collection of Kubernetes resources that compose a release as a single unit, either to install, upgrade, or remove.

At its core, the value that Helm brings to the table – at least for us – is allowing us to templatize our experience with Kubernetes resources, providing a standard interface for operators or high-level software orchestrators to control the installation and life cycle of Kubernetes applications.

OpenStack-Helm¶

The OpenStack-Helm project provides a framework to enable the deployment, maintenance, and upgrading of loosely coupled OpenStack services and their dependencies individually or as part of complex environments.

OpenStack-Helm is essentially a marriage of Kubernetes, Helm, and OpenStack, and seeks to create Helm charts for each OpenStack service. These Helm charts provide complete life cycle management for these OpenStack services.

Users of OpenStack-Helm either deploy all or individual OpenStack components along with their required dependencies. It heavily borrows concepts from Stackanetes and complex Helm application deployments. Ideally, at the end of the day, this project is meant to be a collaborative project that brings OpenStack applications into a cloud-native model.

Divingbell¶

Divingbell is a lightweight solution for:

Bare metal configuration management for a few very targeted use cases
Bare metal package manager orchestration

What problems does it solve?¶

The needs identified for Divingbell were:

To plug gaps in day 1 tools (e.g., Drydock) for node configuration
To provide a day 2 solution for managing these configurations going forward
[Future] To provide a day 2 solution for system level host patching

Berth¶

Berth is a deliberately minimalist VM runner for Kubernetes.

Process Flows¶

Site Authoring and Deployment Guide¶

The document contains the instructions for standing up a greenfield Airship site. This can be broken down into two high-level pieces:

Site authoring guide(s): Describes how to craft site manifests and configs required to perform a deployment. The primary site authoring guide is for deploying Airship sites, where OpenStack is the target platform deployed on top of Airship.
Deployment guide(s): Describes how to apply site manifests for a given site.

This document is an “all in one” site authoring guide + deployment guide for a standard Airship deployment. For the most part, the site authoring guidance lives within airship-seaworthy reference site in the form of YAML comments.

Terminology¶

Cloud: A platform that provides a standard set of interfaces for IaaS consumers.

OSH: (OpenStack Helm) is a collection of Helm charts used to deploy OpenStack on kubernetes.

Undercloud/Overcloud: Terms used to distinguish which cloud is deployed on top of the other. In Airship sites, OpenStack (overcloud) is deployed on top of Kubernetes (undercloud).

Airship: A specific implementation of OpenStack Helm charts onto kubernetes, the deployment of which is the primary focus of this document.

Control Plane: From the point of view of the cloud service provider, the control plane refers to the set of resources (hardware, network, storage, etc) sourced to run cloud services.

Data Plane: From the point of view of the cloud service provider, the data plane is the set of resources (hardware, network, storage, etc.) sourced to run consumer workloads. When used in this document, “data plane” refers to the data plane of the overcloud (OSH).

Host Profile: A host profile is a standard way of configuring a bare metal host. Encompasses items such as the number of bonds, bond slaves, physical storage mapping and partitioning, and kernel parameters.

Component Overview¶

Node Overview¶

This document refers to several types of nodes, which vary in their purpose, and to some degree in their orchestration / setup:

Build node: This refers to the environment where configuration documents are built for your environment (e.g., your laptop)
Genesis node: The “genesis” or “seed node” refers to a node used to get a new deployment off the ground, and is the first node built in a new deployment environment.
Control / Controller nodes: The nodes that make up the control plane. (Note that the Genesis node will be one of the controller nodes.)
Compute nodes / Worker Nodes: The nodes that make up the data plane

Support¶

Bugs may be viewed and reported at the following locations, depending on the component:

OpenStack Helm: OpenStack Storyboard group
Airship: Bugs may be filed using OpenStack Storyboard for specific projects in Airship group:

Hardware Prep¶

Disk¶

For servers that are in the control plane (including Genesis):
- Two-disk RAID-1: Operating System
- Two disks JBOD: Ceph Journal/Meta for control plane
- Remaining disks JBOD: Ceph OSD for control plane
For servers that are in the tenant data plane (compute nodes):
- Two-disk RAID-1: Operating System
- Two disks JBOD: Ceph Journal/Meta for tenant-ceph
- Two disks JBOD: Ceph OSD for tenant-ceph
- Remaining disks need to be configured according to the host profile target for each given server (e.g. RAID-10 for OpenStack Ephemeral).

BIOS and IPMI¶

Virtualization enabled in BIOS
IPMI enabled in server BIOS (e.g., IPMI over LAN option enabled)
IPMI IPs assigned, and routed to the environment you will deploy into Note: Firmware bugs related to IPMI are common. Ensure you are running the latest firmware version for your hardware. Otherwise, it is recommended to perform an iLo/iDrac reset, as IPMI bugs with long-running firmware are not uncommon.
Set PXE as first boot device and ensure the correct NIC is selected for PXE

Network¶

You have a network you can successfully PXE boot with your network topology and bonding settings (dedicated PXE interace on untagged/native VLAN in this example)
You have (VLAN) segmented, routed networks accessible by all nodes for:
1. Management network(s) (k8s control channel)
2. Calico network(s)
3. Storage network(s)
4. Overlay network(s)
5. Public network(s)

HW Sizing and minimum requirements¶

Node	disk	memory	cpu
Build	10 GB	4 GB	1
Genesis	100 GB	16 GB	8
Control	10 TB	128 GB	24
Compute	N/A*	N/A*	N/A*

Workload driven (determined by host profile)

Establishing build node environment¶

On the machine you wish to use to generate deployment files, install required tooling:
```
sudo apt -y install docker.io git
```

Clone and link the required git repos as follows:

git clone https://git.openstack.org/openstack/airship-pegleg
git clone https://git.openstack.org/openstack/airship-treasuremap

Building Site documents¶

This section goes over how to put together site documents according to your specific environment, and generate the initial Promenade bundle needed to start the site deployment.

Preparing deployment documents¶

In its current form, pegleg provides an organized structure for YAML elements, in order to separate common site elements (i.e., global folder) from unique site elements (i.e., site folder).

To gain a full understanding of the pegleg structure, it is highly recommended to read pegleg documentation on this here.

The airship-seaworthy site may be used as reference site. It is the principal pipeline for integration and continuous deployment testing of Airship.

Change directory to the airship-treasuremap/site folder and copy the airship-seaworthy site as follows:

NEW_SITE=mySite # replace with the name of your site
cd airship-treasuremap/site
cp -r airship-seaworthy $NEW_SITE

Remove airship-seaworthy specific certificates.

rm -f airship-treasuremap/site/${NEW_SITE}/secrets/certificates/certificates.yaml

You will then need to manually make changes to these files. These site manifests are heavily commented to explain parameters, and importantly identify all of the parameters that need to change when authoring a new site.

These areas which must be updated for a new site are flagged with the label NEWSITE-CHANGEME in YAML commentary. Search for all instances of NEWSITE-CHANGEME in your new site definition, and follow the instructions that accompany the tag in order to make all needed changes to author your new Airship site.

Because some files depend on (or will repeat) information from others, the order in which you should build your site files is as follows:

site/$NEW_SITE/networks/physical/networks.yaml
site/$NEW_SITE/baremetal/nodes.yaml
site/$NEW_SITE/networks/common-addresses.yaml
site/$NEW_SITE/pki/pki-catalog.yaml
All other site files

Register DNS names¶

Register the following list of DNS names:

cloudformation.DOMAIN
compute.DOMAIN
dashboard.DOMAIN
grafana.DOMAIN
iam.DOMAIN
identity.DOMAIN
image.DOMAIN
kibana.DOMAIN
nagios.DOMAIN
network.DOMAIN
nova-novncproxy.DOMAIN
object-store.DOMAIN
orchestration.DOMAIN
placement.DOMAIN
shipyard.DOMAIN
volume.DOMAIN

Here DOMAIN is a name of ingress domain, you can find it in the data.dns.ingress_domain section of site/${NEW_SITE}/secrets/certificates/ingress.yaml configuration file.

Run the following command to get up to date list of required DNS names:

grep -E 'host: .+DOMAIN' site/${NEW_SITE}/software/config/endpoints.yaml | \
    sort -u | awk '{print $2}'

Update Secrets¶

Replace passphrases under site/${NEW_SITE}/secrets/passphrases/ with random generated ones:

Passpharses generation openssl rand -hex 10
UUID generation uuidgen (e.g. for Ceph filesystem ID)
Update secrets/passphrases/ipmi_admin_password.yaml with IPMI password
Update secrets/passphrases/ubuntu_crypt_password.yaml with password hash:

python3 -c "from crypt import *; print(crypt('<YOUR_PASSWORD>', METHOD_SHA512))"

Configure certificates in site/${NEW_SITE}/secrets/certificates/ingress.yaml, they need to be issued for the domains configured in Register DNS names section.

Caution

It is required to configure valid certificates, self-signed certificates are not supported.

Control Plane & Tenant Ceph Cluster Notes¶

Configuration variables for ceph control plane are located in:

site/${NEW_SITE}/software/charts/ucp/ceph/ceph-osd.yaml
site/${NEW_SITE}/software/charts/ucp/ceph/ceph-client.yaml

Configuration variables for tenant ceph are located in:

site/${NEW_SITE}/software/charts/osh/openstack-tenant-ceph/ceph-osd.yaml
site/${NEW_SITE}/software/charts/osh/openstack-tenant-ceph/ceph-client.yaml

Setting highlights:

data/values/conf/storage/osd[*]/data/location: The block device that will be formatted by the Ceph chart and used as a Ceph OSD disk
data/values/conf/storage/osd[*]/journal/location: The block device backing the ceph journal used by this OSD. Refer to the journal paradigm below.
data/values/conf/pool/target/osd: Number of OSD disks on each node

Assumptions:

Ceph OSD disks are not configured for any type of RAID, they are configured as JBOD when connected through a RAID controller. If RAID controller does not support JBOD, put each disk in its own RAID-0 and enable RAID cache and write-back cache if the RAID controller supports it.
Ceph disk mapping, disk layout, journal and OSD setup is the same across Ceph nodes, with only their role differing. Out of the 4 control plane nodes, we expect to have 3 actively participating in the Ceph quorom, and the remaining 1 node designated as a standby Ceph node which uses a different control plane profile (cp_*-secondary) than the other three (cp_*-primary).
If doing a fresh install, disk are unlabeled or not labeled from a previous Ceph install, so that Ceph chart will not fail disk initialization.

It’s highly recommended to use SSD devices for Ceph Journal partitions.

If you have an operating system available on the target hardware, you can determine HDD and SSD devices with:

lsblk -d -o name,rota

where a rota (rotational) value of 1 indicates a spinning HDD, and where a value of 0 indicates non-spinning disk (i.e. SSD). (Note - Some SSDs still report a value of 1, so it is best to go by your server specifications).

For OSDs, pass in the whole block device (e.g., /dev/sdd), and the Ceph chart will take care of disk partitioning, formatting, mounting, etc.

For Ceph Journals, you can pass in a specific partition (e.g., /dev/sdb1), note that it’s not required to pre-create these partitions, Ceph chart will create journal partitions automatically if they don’t exist. By default the size of every journal partition is 10G, make sure there is enough space available to allocate all journal partitions.

Consider the following example where:

/dev/sda is an operating system RAID-1 device (SSDs for OS root)
/dev/sd[bc] are SSDs for ceph journals
/dev/sd[efgh] are HDDs for OSDs

The data section of this file would look like:

data:
  values:
    conf:
      storage:
        osd:
          - data:
              type: block-logical
              location: /dev/sde
            journal:
              type: block-logical
              location: /dev/sdb1
          - data:
              type: block-logical
              location: /dev/sdf
            journal:
              type: block-logical
              location: /dev/sdb2
          - data:
              type: block-logical
              location: /dev/sdg
            journal:
              type: block-logical
              location: /dev/sdc1
          - data:
              type: block-logical
              location: /dev/sdh
            journal:
              type: block-logical
              location: /dev/sdc2

Manifest linting and combining layers¶

After constituent YAML configurations are finalized, use Pegleg to lint your manifests, and resolve any issues that result from linting before proceeding:

sudo airship-pegleg/tools/pegleg.sh repo \
  -r airship-treasuremap lint

Note: P001 and P003 linting errors are expected for missing certificates, as they are not generated until the next section. You may suppress these warnings by appending -x P001 -x P003 to the lint command.

Next, use pegleg to perform the merge that will yield the combined global + site type + site YAML:

sudo sh airship-pegleg/tools/pegleg.sh site \
  -r airship-treasuremap \
  collect $NEW_SITE

Perform a visual inspection of the output. If any errors are discovered, you may fix your manifests and re-run the lint and collect commands.

After you have an error-free output, save the resulting YAML as follows:

sudo airship-pegleg/tools/pegleg.sh site \
  -r airship-treasuremap \
  collect $NEW_SITE -s ${NEW_SITE}_collected

It is this output which will be used in subsequent steps.

Lastly, you should also perform a render on the documents. The resulting render from Pegleg will not be used as input in subsequent steps, but is useful for understanding what the document will look like once Deckhand has performed all substitutions, replacements, etc. This is also useful for troubleshooting, and addressing any Deckhand errors prior to submitting via Shipyard:

sudo airship-pegleg/tools/pegleg.sh site \
  -r airship-treasuremap \
  render $NEW_SITE

Inspect the rendered document for any errors. If there are errors, address them in your manifests and re-run this section of the document.

Building the Promenade bundle¶

Clone the Promenade repo, if not already cloned:

git clone https://opendev.org/airship/promenade

Refer to the data/charts/ucp/promenade/reference field in airship-treasuremap/global/software/config/versions.yaml. If this is a pinned reference (i.e., any reference that’s not master), then you should checkout the same version of the Promenade repository. For example, if the Promenade reference was 86c3c11... in the versions file, checkout the same version of the Promenade repo which was cloned previously:

(cd promenade && git checkout 86c3c11)

Likewise, before running the simple-deployment.sh script, you should refer to the data/images/ucp/promenade/promenade field in ~/airship-treasuremap/global/software/config/versions.yaml. If there is a pinned reference (i.e., any image reference that’s not latest), then this reference should be used to set the IMAGE_PROMENADE environment variable. For example, if the Promenade image was pinned to quay.io/airshipit/promenade:d30397f... in the versions file, then export the previously mentioned environment variable like so:

export IMAGE_PROMENADE=quay.io/airshipit/promenade:d30397f...

Now, create an output directory for Promenade bundles and run the simple-deployment.sh script:

mkdir ${NEW_SITE}_bundle
sudo -E promenade/tools/simple-deployment.sh ${NEW_SITE}_collected ${NEW_SITE}_bundle

Estimated runtime: About 1 minute

After the bundle has been successfully created, copy the generated certificates into the security folder. Ex:

mkdir -p airship-treasuremap/site/${NEW_SITE}/secrets/certificates
sudo cp ${NEW_SITE}_bundle/certificates.yaml \
  airship-treasuremap/site/${NEW_SITE}/secrets/certificates/certificates.yaml

Regenerate collected YAML files to include copied certificates:

sudo airship-pegleg/tools/pegleg.sh site \
  -r airship-treasuremap \
  collect $NEW_SITE -s ${NEW_SITE}_collected

Genesis node¶

Initial setup¶

Before starting, ensure that the BIOS and IPMI settings match those stated previously in this document. Also ensure that the hardware RAID is setup for this node per the control plane disk configuration stated previously in this document.

Then, start with a manual install of Ubuntu 16.04 on the node you wish to use to seed the rest of your environment standard Ubuntu ISO. Ensure to select the following:

UTC timezone
Hostname that matches the Genesis hostname given in /data/genesis/hostname in airship-treasuremap/site/${NEW_SITE}/networks/common-addresses.yaml.
At the Partition Disks screen, select Manual so that you can setup the same disk partitioning scheme used on the other control plane nodes that will be deployed by MaaS. Select the first logical device that corresponds to one of the RAID-1 arrays already setup in the hardware controller. On this device, setup partitions matching those defined for the bootdisk in your control plane host profile found in airship-treasuremap/site/${NEW_SITE}/profiles/host. (e.g., 30G for /, 1G for /boot, 100G for /var/log, and all remaining storage for /var). Note that the volume size syntax looking like >300g in Drydock means that all remaining disk space is allocated to this volume, and that volume needs to be at least 300G in size.
Ensure that OpenSSH and Docker (Docker is needed because of miniMirror) are included as installed packages
When you get to the prompt, “How do you want to manage upgrades on this system?”, choose “No automatic updates” so that packages are only updated at the time of our choosing (e.g. maintenance windows).
Ensure the grub bootloader is also installed to the same logical device as in the previous step (this should be default behavior).

After installation, ensure the host has outbound internet access and can resolve public DNS entries (e.g., nslookup google.com, curl https://www.google.com).

Ensure that the deployed Genesis hostname matches the hostname in data/genesis/hostname in airship-treasuremap/site/${NEW_SITE}/networks/common-addresses.yaml. If it does not match, then either change the hostname of the node to match the configuration documents, or re-generate the configuration with the correct hostname. In order to change the hostname of the deployed node, you may run the following:

sudo hostname $NEW_HOSTNAME
sudo sh -c "echo $NEW_HOSTNAME > /etc/hostname"
sudo vi /etc/hosts # Anywhere the old hostname appears in the file, replace
                   # with the new hostname

Or to regenerate manifests, re-run the previous two sections with the after updating the genesis hostname in the site definition.

Installing matching kernel version¶

Install the same kernel version on the Genesis host that MaaS will use to deploy new baremetal nodes.

In order to do this, first you must determine the kernel version that will be deployed to those nodes. Start by looking at the host profile definition used to deploy other control plane nodes by searching for control-plane: enabled. Most likely this will be a file under global/profiles/host. In this file, find the kernel info - e.g.:

platform:
  image: 'xenial'
  kernel: 'hwe-16.04'
  kernel_params:
    kernel_package: 'linux-image-4.15.0-34-generic'

In this example, the kernel version is 4.15.0-34-generic. Define any proxy environment variables needed for your environment to reach public ubuntu package repos, and install the matching kernel on the Genesis host (make sure to run on Genesis host, not on the build host):

sudo apt -y install linux-modules-4.15.0-34-generic \
   linux-modules-extra-4.15.0-34-generic \
   linux-image-4.15.0-34-generic linux-headers-4.15.0-34-generic \
   linux-headers-4.15.0-34

Check the installed packages on the genesis host with dpkg --list. If there are any later kernel versions installed, remove them with sudo apt remove, so that the newly install kernel is the latest available.

Install ntpdate/ntp¶

Install and run ntpdate, to ensure a reasonably sane time on genesis host before proceeding:

sudo apt -y install ntpdate
sudo ntpdate ntp.ubuntu.com

If your network policy does not allow time sync with external time sources, specify a local NTP server instead of using ntp.ubuntu.com.

Then, install the NTP client:

sudo apt -y install ntp

Add the list of NTP servers specified in data/ntp/servers_joined in file airship-treasuremap/site/${NEW_SITE}/networks/common-address.yaml to /etc/ntp.conf as follows:

pool NTP_SERVER1 iburst
pool NTP_SERVER2 iburst
(repeat for each NTP server with correct NTP IP or FQDN)

Then, restart the NTP service:

sudo service ntp restart

If you cannot get good time to your selected time servers, consider using alternate time sources for your deployment.

Disable the apparmor profile for ntpd:

sudo ln -s /etc/apparmor.d/usr.sbin.ntpd /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.ntpd

This prevents an issue with the MaaS containers, which otherwise get permission denied errors from apparmor when the MaaS container tries to leverage libc6 for /bin/sh when MaaS container ntpd is forcefully disabled.

Promenade bootstrap¶

Copy the ${NEW_SITE}_bundle and ${NEW_SITE}_collected directories from the build node to the genesis node, into the home directory of the user there (e.g., /home/ubuntu). Then, run the following script as sudo on the genesis node:

cd ${NEW_SITE}_bundle
sudo ./genesis.sh

Estimated runtime: 40m

Following completion, run the validate-genesis.sh script to ensure correct provisioning of the genesis node:

cd ${NEW_SITE}_bundle
sudo ./validate-genesis.sh

Estimated runtime: 2m

Deploy Site with Shipyard¶

Start by cloning the shipyard repository to the Genesis node:

git clone https://opendev.org/airship/shipyard

Refer to the data/charts/ucp/shipyard/reference field in airship-treasuremap/global/software/config/versions.yaml. If this is a pinned reference (i.e., any reference that’s not master), then you should checkout the same version of the Shipyard repository. For example, if the Shipyard reference was 7046ad3... in the versions file, checkout the same version of the Shipyard repo which was cloned previously:

(cd shipyard && git checkout 7046ad3)

Likewise, before running the deckhand_load_yaml.sh script, you should refer to the data/images/ucp/shipyard/shipyard field in airship-treasuremap/global/software/config/versions.yaml. If there is a pinned reference (i.e., any image reference that’s not latest), then this reference should be used to set the SHIPYARD_IMAGE environment variable. For example, if the Shipyard image was pinned to quay.io/airshipit/shipyard@sha256:dfc25e1... in the versions file, then export the previously mentioned environment variable:

export SHIPYARD_IMAGE=quay.io/airshipit/shipyard@sha256:dfc25e1...

Export valid login credentials for one of the Airship Keystone users defined for the site. Currently there is no authorization checks in place, so the credentials for any of the site-defined users will work. For example, we can use the shipyard user, with the password that was defined in airship-treasuremap/site/${NEW_SITE}/secrets/passphrases/ucp_shipyard_keystone_password.yaml. Ex:

export OS_USERNAME=shipyard
export OS_PASSWORD=46a75e4...

(Note: Default auth variables are defined here, and should otherwise be correct, barring any customizations of these site parameters).

Next, run the deckhand_load_yaml.sh script providing an absolute path to a directory that contains collected manifests:

sudo -E shipyard/tools/deckhand_load_yaml.sh ${NEW_SITE} $(pwd)/${NEW_SITE}_collected

Estimated runtime: 3m

Now deploy the site with shipyard:

cd shipyard/tools/
sudo -E ./deploy_site.sh

Estimated runtime: 1h30m

The message Site Successfully Deployed is the expected output at the end of a successful deployment. In this example, this means that Airship and OSH should be fully deployed.

Disable password-based login on Genesis¶

Before proceeding, verify that your SSH access to the Genesis node is working with your SSH key (i.e., not using password-based authentication).

Then, disable password-based SSH authentication on Genesis in /etc/ssh/sshd_config by uncommenting the PasswordAuthentication and setting its value to no. Ex:

PasswordAuthentication no

Then, restart the ssh service:

sudo systemctl restart ssh

Configuration Update Guide¶

The guide contains the instructions for updating the configuration of a deployed Airship environment. Please refer to Site Authoring and Deployment Guide if you do not have an Airship environment already deployed.

Update of an Airship environment consists of the following stages:

Prepare the configuration: before deploying any changes, a user should prepare and validate the manifests on a build node using Airship Pegleg.
Deploy the changes: during this stage, a user uploads the configuration to the Airship environment and starts the deployment using Airship Shipyard.

Note

This guide assumes you have Airship Pegleg and Airship Shipyard tools installed and configured; please refer to Site Authoring and Deployment Guide for the details.

Configuring Airship CLI¶

Clone the Airship Treasuremap repository and switch to correct version.

git clone https://opendev.org/airship/treasuremap
cd treasuremap/
# List available tags.
git tag --list
# Switch to the version your site is using.
git checkout {your-tag}
# Go back to a previous directory.
cd ..

Configure environment variables with the name of your site, and specify a path to the directory where site configuration is stored; for this example, we use Airship Seaworthy site:

export SITE=airship-seaworthy
export SITE_PATH=treasuremap/site/airship-seaworthy

Updating the manifests¶

Changing the configuration consists of the following steps:

Change site manifests.
Lint the manifests.
Collect the manifests.
Copy the manifests to the Airship environment.

Linting and collecting the manifests is done using Airship Pegleg.

For this example, we are going to update a debug level for keystone logs in a site layer.

Note

It is also possible to update the configuration in a global layer; for more details on Airship layering mechanism see Pegleg Definition Artifact Layout documentation.

Create an override file ${SITE_PATH}/software/charts/osh/openstack-keystone/keystone.yaml with the following content:

---
schema: armada/Chart/v1
metadata:
  schema: metadata/Document/v1
  name: keystone
  replacement: true
  layeringDefinition:
    abstract: false
    layer: site
    parentSelector:
      name: keystone-global
    actions:
      - method: merge
        path: .
  storagePolicy: cleartext
data:
  values:
    conf:
      logging:
        logger_keystone:
          level: DEBUG
...

Check that the configuration is valid:

sudo ./treasuremap/tools/airship pegleg site -r treasuremap/ \
  lint ${SITE}

Collect the configuration:

sudo ./treasuremap/tools/airship pegleg site \
  -r treasuremap/ collect $SITE -s ${SITE}_collected

Copy the configuration to a node that has the access to the site’s Shipyard API, if the current node does not; this node can be one of your controllers:

scp -r ${SITE}_collected {genesis-ip}:/home/{user-name}/${SITE}_collected

Deploying the changes¶

After you copied the manifests, there are just a few steps needed to start the deployment:

Upload the changes to Airship Deckhand.
Start the deployment using Airship Shipyard.

Install Airship CLI as described in Configuring Airship CLI section.

Set the name of your site:

::: export SITE=airship-seaworthy

Configure credentials for accessing Shipyard; the password is stored in ucp_shipyard_keystone_password secret, you can find it in site/airship-seaworthy/secrets/passphrases/ucp_shipyard_keystone_password.yaml configuration file of your site.

export OS_USERNAME=shipyard
export OS_PASSWORD={shipyard_password}

Upload the changes to Airship Deckhand:

# Upload the configuration.
sudo -E ./treasuremap/tools/airship shipyard \
    create configdocs ${SITE} --replace --directory=${SITE}_collected

# Commit the configuration.
sudo -E ./treasuremap/tools/airship shipyard commit configdocs

Run the deployment:

sudo -E ./treasuremap/tools/airship shipyard create action update_site

You can also run update_software instead of update_site which skips hardware configuration and only applies the changes to services that are running on top of Kubernetes.

Now you can track the deployment progress using the following commands:

# Get all actions that were executed on you environment.
sudo -E ./treasuremap/tools/airship shipyard get actions

# Show all the steps within the action.
sudo -E ./treasuremap/tools/airship shipyard describe action/{action_id}

All steps will have status success when the update finishes.

Troubleshooting Guide¶

This guide provides information on troubleshooting of an Airship environment. Debugging of any software component starts with gathering more information about the failure, so the intention of the document is not to describe specific issues that one can encounter, but to provide a generic set of instructions that a user can follow to find the root cause of the problem.

For additional support you can contact the Airship team via IRC or mailing list, use Airship bug tracker to search and create issues.

Configuring Airship CLI¶

Many commands from this guide use Airship CLI, this section describes how to get it configured on your environment.

git clone https://opendev.org/airship/treasuremap
cd treasuremap/
# List available tags.
git tag --list
# Switch to the version your site is using.
git checkout {your-tag}
# Go back to a previous directory.
cd ..
# Run it without arguments to get a help message.
sudo ./treasuremap/tools/airship

Manifests Preparation¶

When you do any configuration changes to the manifests, there are a few commands that you can use to validate the changes without uploading them to the Airship environment.

Run lint command for your site; it helps to catch the errors related to documents duplication, broken references, etc.

Example:

sudo ./treasuremap/tools/airship pegleg site -r airship-treasuremap/ \
    lint {site-name}

If you create configuration overrides or do changes to substitutions, it is recommended to run render command this command merges the layers and renders all substitutions. This allows finding what parameters are passed to Helm as overrides for Charts’ defaults.

Example:

# Saves the result into rendered.txt file.
sudo ./treasuremap/tools/airship pegleg site -r treasuremap/ \
    render -o rendered.txt ${SITE}

Deployment Failure¶

During the deployment, it is important to identify a specific step where it fails, there are two major deployment steps:

Drydock build: deploys Operating System.
Armada build: deploys Helm Charts.

After Configuring Airship CLI, setup credentials for accessing Shipyard; the password is stored in ucp_shipyard_keystone_password secret, you can find it in site/airship-seaworthy/secrets/passphrases/ucp_shipyard_keystone_password.yaml configuration file of your site.

export OS_USERNAME=shipyard
export OS_PASSWORD={shipyard_password}

Now you can use the following commands to access Shipyard:

# Get all actions that were executed on you environment.
sudo ./treasuremap/tools/airship shipyard get actions
# Show all the steps within the action.
sudo ./treasuremap/tools/airship shipyard describe action/{action_id}
# Get a bit more details on the step.
sudo ./treasuremap/tools/airship shipyard describe step/{action_id}/armada_build
# Print the logs from the step.
sudo ./treasuremap/tools/airship shipyard logs step/{action_id}/armada_build

After the failed step is determined, you can access the logs of a specific service (e.g., drydock-api/maas or armada-api) to get more information on the failure, note that there may be multiple pods of a single service running, you need to check all of them to find where the most recent logs are available.

Example of accessing Armada API logs:

# Get all pods running on the cluster and find a name of the pod you are
# interested in.
kubectl get pods -o wide --all-namespaces

# See the logs of specific pod.
kubectl logs -n ucp -f --tail 200 armada-api-d5f757d5-6z6nv

In some cases you want to restart your pod, there is no dedicated command for that in Kubernetes. However, you can delete the pod, it will be restarted by Kubernetes to satisfy replication factor.

# Restart Armada API service.
kubectl delete pod -n ucp armada-api-d5f757d5-6z6nv

Ceph¶

Many stateful services in Airship rely on Ceph to function correctly. For more information on Ceph debugging follow an official Ceph debugging guide.

Although Ceph tolerates failures of multiple OSDs, it is important to make sure that your Ceph cluster is healthy.

Example:

# Get a name of Ceph Monitor pod.
CEPH_MON=$(sudo kubectl get pods --all-namespaces -o=name | \
    grep ceph-mon | sed -n 1p | sed 's|pod/||')
# Get the status of the Ceph cluster.
sudo kubectl exec -it -n ceph ${CEPH_MON} -- ceph -s

Cluster is in a helthy state when health parameter is set to HEALTH_OK.

When the cluster is unhealthy, and some Placement Groups are reported to be in degraded or down states, determine the problem by inspecting the logs of Ceph OSD that is down using kubectl.

# Get a name of Ceph Monitor pod.
CEPH_MON=$(sudo kubectl get pods --all-namespaces -o=name | \
    grep ceph-mon | sed -n 1p | sed 's|pod/||')
# List a hierarchy of OSDs in the cluster to see what OSDs are down.
sudo kubectl exec -it -n ceph ${CEPH_MON} -- ceph osd tree

There are a few other commands that may be useful during the debugging:

# Get a name of Ceph Monitor pod.
CEPH_MON=$(sudo kubectl get pods --all-namespaces -o=name | \
    grep ceph-mon | sed -n 1p | sed 's|pod/||')

# Get a detailed information on the status of every Placement Group.
sudo kubectl exec -it -n ceph ${CEPH_MON} -- ceph pg dump

# List allocated block devices.
sudo kubectl exec -it -n ceph ${CEPH_MON} -- rbd ls
# See what client uses the device.
sudo kubectl exec -it -n ceph ${CEPH_MON} -- rbd status \
    kubernetes-dynamic-pvc-e71e65a9-3b99-11e9-bf31-e65b6238af01

# List all Ceph block devices mounted on a specific host.
mount | grep rbd

Airship Seaworthy¶

Airship Seaworthy is a multi-node site deployment reference and continuous integration pipeline.

The site manifests are available at site/airship-seaworthy.

Pipeline¶

Airship Seaworthy pipeline automates deployment flow documented in Site Authoring and Deployment Guide.

The pipeline is implemented as Jenkins Pipeline (Groovy), see code for the pipeline at Jenkinsfile.

Versions¶

The manifest overrides (versions.yaml) are setup to deploy OpenStack Ocata.

The versions are kept up to date via updater.py, a utility that updates versions.yaml latest charts and (selected) images.

The pipeline attempts to uplift and deploy latest versions on daily bases.

Hardware¶

While HW configuration is flexible, Airship Seaworthy reference manifests reflect full HA deployment, similar to what might be expected in production.

Reducing number of control/compute nodes will require site overrides to align parts of the system such as Ceph replication, etcd, etc.

Airship Seaworthy site has 6 DELL R720xd bare-metal servers: 3 control, and 3 compute nodes. See host profiles for the servers here.

Control (masters)

cab23-r720-11
cab23-r720-12
cab23-r720-13

Compute (workers)

cab23-r720-14
cab23-r720-17
cab23-r720-19

Network¶

Physical (underlay) networks are described in Drydock site configuration here. It defines OOB (iLO/IPMI), untagged PXE, and multiple tagged general use networks.

Calico overlay for k8s POD networking uses IPIP mesh.

BGP peering is supported but not enabled in this setup, see Calico chart.

Airskiff¶

Skiff (n): a shallow, flat-bottomed, open boat
Airskiff (n): a learning development, and gating environment for Airship

What is Airskiff¶

Airskiff is an easy way to get started with the software delivery components of Airship:

Airskiff is packaged with a set of deployment scripts modeled after the OpenStack-Helm project for seamless developer setup.

These scripts:

Download, build, and containerize the Airship components above from source.
Deploy a Kubernetes cluster using Minikube.
Deploy Armada, Deckhand, and Shipyard using the latest Armada image.
Deploy OpenStack using the Airskiff site and charts from the OpenStack-Helm project.

Warning

Airskiff is not safe for production use. These scripts are only intended to deploy a minimal development environment.

Common Deployment Requirements¶

This section covers actions that may be required for some deployment scenarios.

Passwordless sudo¶

Airskiff relies on scripts that utilize the sudo command. Throughout this guide the assumption is that the user is: ubuntu. It is advised to add the following lines to /etc/sudoers:

root    ALL=(ALL) NOPASSWD: ALL
ubuntu  ALL=(ALL) NOPASSWD: ALL

Proxy Configuration¶

Note

This section assumes you have properly defined the standard http_proxy, https_proxy, and no_proxy environment variables and have followed the Docker proxy guide to create a systemd drop-in unit.

In order to deploy Airskiff behind proxy servers, define the following environment variables:

export USE_PROXY=true
export PROXY=${http_proxy}
export no_proxy=${no_proxy},10.0.2.15,.svc.cluster.local
export NO_PROXY=${NO_PROXY},10.0.2.15,.svc.cluster.local

Note

The .svc.cluster.local address is required to allow the OpenStack client to communicate without being routed through proxy servers. The IP address 10.0.2.15 is the advertised IP address of the minikube Kubernetes cluster. Replace the addresses if your configuration does not match the one defined above.

Deploy Airskiff¶

Deploy Airskiff using the deployment scripts contained in the tools/deployment/airskiff directory of the airship-treasuremap repository.

Note

Scripts should be run from the root of airship-treasuremap repository.