Included resource types

A collection of resource types come pre-installed on the OpsChain step runner image, this guide covers what they are and how to use them.

Resource type summary

The table below outlines the file to require in your resource definition and the resource types that will become available.

Require	Resource type	Description
none	artefact	Load a stored artefact as a prerequisite of another action
none	artefact_load	Load a stored artefact automatically, for the whole file to use
opschain-chef	chef_cookbook	Upload cookbooks to a Chef server
	chef_environment	Pin cookbook versions on, and update, a Chef environment object
	chef_node	Set the run list of one or more Chef nodes
none	git_clone	Check out one of the project's configured Git remotes into the step's working environment
opschain-kubernetes	kubernetes_resource	Manage Kubernetes resources via manifests in your project repo
	kubernetes_daemonset	Perform common operations on a Kubernetes daemonset resource
	kubernetes_deployment	Perform common operations on a Kubernetes deployment resource
	kubernetes_statefulset	Perform common operations on a Kubernetes statefulset resource
opschain-ssh-key-pair	ssh_key_pair	Generate an SSH public/private key pair and optionally stores the key files in OpsChain properties
opschain-terraform	terraform_config	Exposes the RubyTerraform Gem

Usage

The resource types are pre-installed in the OpsChain step runner image via the opschain-resource-types Gem. To use them, add a require to your actions.rb (or wherever you define your resources):

# replace 'opschain-infrastructure' with the relevant value from the "Require" column in the table above
require 'opschain-infrastructure'

# replace transport_factory with the required resource type from the "Resource Type" column in the table above
transport_factory :my_transport_factory do
  ...
end

Projects with their own Gemfile

If your project repository has a Gemfile, OpsChain installs the gems it lists together with every gem pre-installed on the runner image. Your actions can require opschain-resource-types, the MintPress gems and the other pre-installed gems without listing them in your Gemfile, and the versions pinned in your Gemfile.lock are kept.

If a version your Gemfile pins conflicts with a pre-installed gem's own dependencies — activesupport ~> 6.1, for example — OpsChain installs your Gemfile on its own instead and says so in the image build log. Only the gems your Gemfile lists can then be loaded, so add any pre-installed gem your actions need to it.

A custom runner image gets this behaviour from the bundle_and_lint.sh step in the sample Dockerfile. If your Dockerfile runs bundle install itself, replace it with opschain-exec opschain-bundle-install.

OpsChain infrastructure

Requiring opschain-infrastructure currently provides a minimal set of resource types for the Confluent OpsChain example project. More support will be added over time.

OpsChain artefacts

The artefact and artefact_load resource types are both loaded before your actions.rb is read, so they need no require of their own - same as git_clone. Both are thin resource-type wrappers around the OpsChain.artefact.store/.load calls described in full, with worked examples, in artefacts — this section covers only their resource-type properties and actions; read that guide first for what an artefact is and how store/load/owner:/labels: work.

Resource type properties - Artefacts

The artefact resource type accepts:

Property	Default value	Description
code	the resource's own name	The artefact's code, as passed to OpsChain.artefact.store/.load — see artefacts. artefact :war do code 'war' end can just be artefact :war, since the two usually match - only needed explicitly when they don't.
owner	(none)	Reach a different project, environment or asset — see choosing where an artefact belongs.
labels	[]	Narrow load to a specific stored version — see keeping more than one version.
path	(none)	Where to write the loaded artefact — see loading an artefact. Omit to let OpsChain choose.

The artefact_load resource type accepts:

Property	Default value	Description
code	the resource's own name	The artefact's code to load - defaults to matching artefact/artefact_load's own behaviour, see above.
owner	(none)	Reach a different project, environment or asset — see choosing where an artefact belongs.
labels	[]	Narrow load to a specific stored version — see keeping more than one version.
path	(none)	Where to write the loaded artefact — see loading an artefact. Omit to let OpsChain choose.
load_during_dry_run	false	Whether to load the artefact when OpsChain loads your actions to discover them rather than to run a step — see dry runs.

The same git_clone gotchas apply here

artefact_load loads as soon as your actions are read, exactly like git_clone checking out a repository, so the same three things documented under when the repository is checked out apply equally: declare it before any code that reads the loaded file, share one declaration across several templates in the same repository if more than one needs it, and — the one that bites hardest — set load_during_dry_run on its own, not through the bulk properties form, which silently does nothing during a dry run for exactly the same reason clone_during_dry_run does.

Actions - Artefacts

artefact provides a single load action, referenced as a prerequisite of the action that needs it — it does not expose store as an action, since a stored artefact must always come from the same action that produced the underlying file:

artefact :war

action deploy_war: ['war:load'] do
  system('systemctl restart tomcat')
end

code defaults to the resource's own name (war above), so it needs no block at all here. Give it an explicit code (or any of the other properties) with the usual block form when the resource's name and the artefact's code differ.

artefact_load has no actions to invoke — declaring it with on_definition_action :load loads the artefact as soon as your actions are read, for the whole file to use. Here the resource's name and the artefact's code differ, so code is still declared explicitly:

artefact_load :deploy_config do
  code 'deploy-config-yaml'
end

OpsChain Chef

Requiring opschain-chef provides three resource types that manage Chef Infra Server objects through knife: chef_cookbook, which uploads cookbooks to the Chef server, chef_environment, which pins the cookbook versions an environment runs and writes the environment object as a whole, and chef_node, which sets the run list of one or more nodes.

Between them they decide what the next chef-client run on a host does: the cookbook has to be on the Chef server, the environment's cookbook_versions constraints say which version of it is resolved, and the node's run list says which recipes run. Upload, pin and set the run list before the hosts run, not after.

Everything runs on the step runner. Nothing runs on a target host, so there is no host property.

A Cinc server is managed the same way. knife speaks the same API to both, and the resource type does not care which is at the other end.

Prerequisites

Unlike the Kubernetes and Terraform resource types, this one needs no custom step runner Dockerfile. The knife gem (18.5.0) is installed in the OpsChain step runner image and is on the PATH.

The knife configuration is not, and every action fails with knife's own configuration error until you supply one. Add it as an OpsChain file property written to /opt/opschain/.chef/config.rb, which is the opschain user's ~/.chef and one of the locations knife searches by itself. If you keep it somewhere else, name that path in the knife_config property:

chef_environment :app_environment do
  environment 'dev'
  knife_config lazy { OpsChain.properties.chef.knife_config_file }
end

Read the path from a property rather than hard coding it, so the same actions can run against a different Chef server per environment.

chef_cookbook

chef_cookbook uploads cookbooks from the step runner to the Chef server.

chef_cookbook :app_cookbooks do
  cookbooks ['app-deployments', 'base']
  cookbook_path 'chef/cookbooks'
  knife_config lazy { OpsChain.properties.chef.knife_config_file }
end
# provides app_cookbooks:show and app_cookbooks:upload actions

The version a cookbook is uploaded as is the one in its own metadata.rb, and is not something this resource type sets. Uploading a cookbook does not change what the hosts run either — use chef_environment to pin which of the uploaded versions an environment resolves.

Choosing the cookbooks

Name each cookbook to upload. cookbooks takes a single name or a list:

chef_cookbook :one_cookbook do
  cookbooks 'app-deployments'
end

There is no option to upload everything in the cookbook path. A step uploads what its resource names, so a stray directory in the checkout cannot add a cookbook to the Chef server, and the change records what was sent.

Names are checked before knife runs, so a path or a versioned entry fails before anything is uploaded:

cookbook name "chef/cookbooks/app-deployments" is not valid: supply the name alone, a Chef cookbook name holds only letters, numbers, underscores and hyphens

chef_environment also has a cookbooks property, and there it is a hash of cookbook name to version. Supplying a hash here fails rather than uploading cookbooks named after the versions in it:

the "cookbooks" property must be a list of cookbook names, not a map: a chef_cookbook uploads whole cookbooks and takes each version from the cookbook's own metadata

Where the cookbooks are read from

cookbook_path is the directory holding the cookbooks — a repository checked out by git_clone, or the directory a preceding berks vendor step wrote to. A relative path is resolved against the step's working directory, /opt/opschain, where your project repository is checked out.

Leave cookbook_path unset to use the cookbook_path from your knife configuration.

Overwriting and freezing versions

knife overwrites a version the Chef server already holds. That suits a cookbook still in development, and does not suit one a production host has already run.

Set freeze_versions true to mark each uploaded version frozen. Uploading that same version again then fails instead of replacing the code the hosts ran, and releasing a further change means bumping the version in metadata.rb. force true overwrites a frozen version, so keep it off the resource that runs routinely.

Set include_dependencies true to upload the cookbooks each named cookbook depends on as well.

Resource type properties - Chef cookbook

The chef_cookbook resource type accepts the following properties:

Property	Default value	Description
cookbooks		The cookbooks to act on, as a single name or a list of names. Both actions fail with A "cookbooks" property is required when it is empty.
cookbook_path		The directory the cookbooks are read from, passed to knife as --cookbook-path. Leave it unset to use the cookbook_path from your knife configuration.
freeze_versions	false	Mark each uploaded version frozen, so uploading that version again fails rather than replacing it. Passed to knife as --freeze.
force	false	Overwrite a version the Chef server holds frozen. Passed to knife as --force.
include_dependencies	false	Upload the cookbooks each named cookbook depends on as well. Passed to knife as --include-dependencies.
ignore_failure	false	Log a warning and return nil rather than raising. For the usual case of wanting the change to carry on, use the ignore_failures DSL instead.
knife_binary	knife	The knife executable to run.
knife_config		Passed to knife as -c. Leave it unset to use knife's own configuration resolution.
store_in		Copy the versions into OpsChain properties: :environment or :project. Nothing is written to properties when this is unset.
store_key	chef.cookbooks	The dotted property path store_in writes to.

Actions

The chef_cookbook resource type provides the following actions:

Action	Description
show	Read and return the versions the Chef server holds for each named cookbook, newest first. Uploads nothing.
upload	Upload each named cookbook, then read and return the versions the Chef server holds for it.

Both return a map keyed by cookbook name, so an action can read what the Chef server now holds:

action :report_cookbook_versions do
  OpsChain.logger.info resource('app_cookbooks').controller.show.inspect
  # => {"app-deployments"=>["30.0.12", "30.0.11"], "base"=>["1.4.0"]}
end

A cookbook the Chef server has never held reads as an empty list rather than failing, so show answers for a cookbook that has not been uploaded yet.

Every upload writes to the Chef server

chef_environment and chef_node read first and write only what differs. chef_cookbook uploads every time.

A cookbook's contents can change without the version in its metadata.rb moving, so the version on the server does not tell you whether it matches the cookbook on disk, and there is nothing to compare against. The versions each action returns are read back from the Chef server after the upload, so they are the ones it left behind.

chef_environment

Pinning cookbook versions

chef_environment :app_environment do
  environment 'dev'
  cookbooks lazy { { 'deployments-cookbook' => OpsChain.properties.release.cookbook_version } }
  knife_config lazy { OpsChain.properties.chef.knife_config_file }
end
# provides app_environment:show, app_environment:pin and app_environment:update actions

Give versions as plain version strings. '30.0.12' is written as the exact constraint = 30.0.12; a value that already carries an operator, such as '~> 30.0', is used as you wrote it.

pin reads the environment, works out the new constraint set and writes it back only when it differs from what is already there, so a change that does not move the pinning writes nothing to the Chef server.

A version that was never uploaded fails on the hosts, not in your change

Pin a version the Chef server does not hold and the pinning itself succeeds. Every chef-client run on a host in that environment then fails to resolve the cookbook, hours later and somewhere you are not looking.

That is why verify_uploaded defaults to true. Before pinning an exact version, OpsChain asks the Chef server for the cookbook and fails the step if that version is not there:

cookbook "app-deployments" version 30.0.12 is not available on the Chef server

Only exact constraints can be checked this way. A range such as ~> 30.0 is left for the Chef server to resolve and is not checked.

Resource type properties - Chef

The chef_environment resource type accepts the following properties:

Property	Default value	Description
environment	the OpsChain environment code	The Chef environment to act on. The default only fits a project where the Chef environment carries the same name as the OpsChain environment, so set it explicitly whenever the two differ. A change running at project level has no environment to fall back on and fails with An "environment" property is required.
cookbooks		The cookbooks pin constrains, as a hash of cookbook name to version. pin fails with A "cookbooks" property is required when it is empty.
definition		The environment object update writes, as a hash. Mutually exclusive with definition_file.
definition_file		Path to a JSON file holding the environment object update writes. A relative path is resolved against the step's working directory, /opt/opschain, where your project repository is checked out. Mutually exclusive with definition.
merge	true	Merge into what the environment already holds rather than replacing it — see merging or replacing what the environment already holds.
verify_uploaded	true	Confirm the Chef server holds an exactly pinned version before pinning it. Applies to pin only.
ignore_failure	false	Log a warning and return nil rather than raising. For the usual case of wanting the change to carry on, use the ignore_failures DSL instead; reach for this property only when later code in the same action has to keep running after knife fails.
knife_binary	knife	The knife executable to run.
knife_config		Passed to knife as -c. Leave it unset to use knife's own configuration resolution.
store_in		Copy the environment object into OpsChain properties: :environment or :project. Nothing is written to properties when this is unset.
store_key	chef.environments.<environment>	The dotted property path store_in writes to.

Actions

The chef_environment resource type provides the following actions:

Action	Description
show	Read the environment object, log it and return it.
pin	Apply the cookbooks constraints to the environment's cookbook_versions, writing only when they differ from the pinning already there.
update	Write definition (or definition_file) to the environment, creating the environment when the Chef server does not hold it yet.

All three return the environment object they read or wrote, so an action can pick values out of it:

action :report_pins do
  versions = resource('app_environment').controller.show['cookbook_versions']
  OpsChain.logger.info versions.inspect
end

Merging or replacing what the environment already holds

merge (default true) decides what happens to the values the environment already carries, and means the matching thing for both actions.

For pin, the cookbooks you name are merged into the existing pinning: a cookbook that is already pinned takes the new version, and every other existing pin is left alone. Set merge false when the versions you supply are the complete set the environment should carry, since any cookbook you do not name then loses its constraint.

For update, the definition is deep merged over the existing environment object, so setting one key under default_attributes leaves its siblings intact. Set merge false to replace the object outright, which is the only way to remove a key.

Updating the environment object

update writes the whole environment object, taken from either a definition hash or the JSON document at definition_file:

chef_environment :app_environment do
  environment 'dev'
  definition_file 'chef/environments/dev.json'
  knife_config lazy { OpsChain.properties.chef.knife_config_file }
end

Supply one or the other. Supplying both fails with The "definition" and "definition_file" properties are mutually exclusive, and supplying neither fails with A "definition" or "definition_file" property is required.

update creates the environment when the Chef server does not hold it yet. pin and show do not, and both fail with Chef environment "dev" does not exist, because an environment missing at pinning time is nearly always a typo in the environment name rather than an environment you meant to create.

The name key is set from the environment property. A definition naming a different environment fails rather than writing to the wrong object:

The supplied definition is named "test" but the environment property is "dev"

Copying the environment into OpsChain properties

The Chef server holds the state. The environment object is copied into OpsChain properties only when store_in asks for it:

chef_environment :app_environment do
  environment 'dev'
  store_in :environment
end

Every action then writes the object it read or wrote to chef.environments.dev in the environment properties. Set store_key to put it somewhere else.

note

Like any modifiable property, the copy is saved on step completion. It is available to subsequent steps and changes, not to the step that wrote it. Reading it back within the same step gives you the value from the previous run.

chef_node

chef_node sets the run list of one or more nodes: the recipes and roles a VM runs, and the order it runs them in.

chef_node :dev_vms do
  search 'chef_environment:dev'
  run_list ['base::bootstrap', 'base::default', 'app::default']
  knife_config lazy { OpsChain.properties.chef.knife_config_file }
end
# provides dev_vms:show and dev_vms:set_run_list actions

Each action reads every target node first and writes only the ones whose run list differs, so a change that does not move a fleet's run list writes nothing to the Chef server.

Choosing the nodes

Name the VMs with nodes, or select them with a Chef search query. Supply one or the other: both together fails with The "nodes" and "search" properties are mutually exclusive, and neither with A "nodes" or "search" property is required.

chef_node :one_vm do
  nodes 'vm1.example.com'
end

chef_node :two_vms do
  nodes ['vm1.example.com', 'vm2.example.com']
end

The resolved node names are logged before anything is written, so the change records which VMs were in scope.

A search with no field name matches VMs you did not ask for

knife rewrites a query holding no : into a fuzzy match across several node fields, so a query that reads like an environment or a hostname matches by substring instead:

tags:*dev* OR roles:*dev* OR fqdn:*dev* OR addresses:*dev* OR policy_name:*dev* OR policy_group:*dev*

The result is a plausible near miss rather than an obvious failure. Measured against one estate, the field query and the bare query returned 46 and 45 nodes: different sets rather than one being a subset of the other, because the bare query also matched the environment name inside fully qualified domain names.

chef_node refuses a query with no : rather than passing it to knife:

the "search" property must be a field query such as chef_environment:dev; knife expands "dev" into a fuzzy match across tags, roles, fqdn, addresses, policy_name and policy_group

A search that matches nothing fails rather than reporting success having touched no VMs, because a mistyped query is otherwise indistinguishable from a fleet that is already correct:

the search "chef_environment:dev" matched no nodes

Set max_nodes to cap how many VMs one action may touch, so a query that matches more of the estate than you meant stops before writing to any of them:

the search "chef_environment:dev" matched 46 nodes, more than the max_nodes limit of 25

Run list entries

Entries are normalised the way Chef parses them, so you can write them the short way:

Written as	Applied as
apache	recipe[apache]
base::bootstrap	recipe[base::bootstrap]
apache@1.2.0	recipe[apache@1.2.0]
recipe[apache]	recipe[apache]
role[web]	role[web]

Normalising what you declare and what the Chef server returns is what makes the action idempotent. Without it run_list ['apache'] would never match the stored recipe[apache], and every change would rewrite every VM in the fleet.

A near miss is rejected before knife is called, rather than being written as a run list entry that only fails when Chef next resolves it:

run list entry "Recipe[apache]" is not valid: expected recipe[name], recipe[name@version] or role[name]
run list entry "apache@1" is not valid: a recipe version must be two or three dot separated numbers, such as apache@1.2 or apache@1.2.0

Replacing or merging the run list

run_list is the complete, ordered list each VM should carry, and set_run_list makes each node match it. Anything the node holds that you do not name is dropped.

Set merge true to append only the entries a node is missing, leaving the order it already holds alone. Replace is the default here, the opposite way round to merge on chef_environment, because a run list is ordered: appending moves where a recipe runs relative to the others, and doing that silently is the more surprising behaviour.

chef_node :dev_vms do
  search 'chef_environment:dev'
  run_list ['audit::default']
  merge true
end

Resource type properties

The chef_node resource type accepts the following properties:

Property	Default value	Description
nodes		The nodes to act on, as a single name or a list of names. Mutually exclusive with search.
search		A Chef search query selecting the nodes to act on, such as chef_environment:dev. It must name a field, for the reason given above. Mutually exclusive with nodes.
run_list		The complete ordered run list the nodes should carry. set_run_list fails with A "run_list" property is required when it is empty.
merge	false	Append the entries a node is missing rather than replacing its run list.
max_nodes		Fail when more than this many nodes resolve, before writing to any of them. Worth setting on any resource that uses search.
ignore_failure	false	Log a warning and return nil rather than raising. For the usual case of wanting the change to carry on, use the ignore_failures DSL instead.
knife_binary	knife	The knife executable to run.
knife_config		Passed to knife as -c. Leave it unset to use knife's own configuration resolution.
store_in		Copy the run lists into OpsChain properties: :environment or :project. Nothing is written to properties when this is unset.
store_key	chef.nodes	The dotted property path store_in writes to. The run lists are stored there as one map keyed by node name, rather than a property path per node, because node names are usually fully qualified domain names and every dot would otherwise become another level of nesting.

Actions

The chef_node resource type provides the following actions:

Action	Description
show	Read and return the run list each target node holds, as a map keyed by node name.
set_run_list	Make each target node's run list match run_list, writing only the nodes that differ.

Why there is no action that writes the whole node

chef_environment has an update action that writes the whole object. chef_node deliberately has no equivalent.

knife node show NODE --format json returns a summary holding only name, chef_environment, run_list and the normal attributes. The node's default, override and automatic attributes appear only in a long listing. Read that summary and write it back with knife node from file and you empty those three attribute trees on the Chef server.

chef_node therefore reads the summary, which always carries the run list, and writes through knife node run_list set, which modifies the node on the server rather than replacing it wholesale.

One consequence: there is no way to empty a run list through this resource type. knife node run_list set requires at least one entry, so an empty run_list is rejected rather than clearing the VM.

OpsChain git clone

The git_clone resource type checks a Git repository out into a directory inside the step's working environment. It is loaded before your actions.rb is read, so it needs no require of its own.

Most often the repository is one of the project's configured Git remotes, referred to by the name the remote was given in OpsChain. By default the resource's name is the remote's name; set git_remote to give the resource a different name, for example to check one remote out more than once. OpsChain checks these out from its own server-side mirror of the remote, so no credentials are needed and the checkout is fast — it never has to talk to the actual remote over the network:

git_clone 'app-config' do
  branch :main
end

A repository that is not configured in the project can be checked out by supplying its URL and credentials instead.

The checkout happens as OpsChain evaluates the git_clone block, so the files are already in place for any code that comes after it — including code that runs while your actions are being loaded. You do not have to invoke anything to make the checkout happen:

git_clone 'app-config' do
  branch :main
end

action :show_config do
  config = YAML.load_file('/opt/opschain/app-config/config.yaml')
end

Resource type properties - Git clone

The git_clone resource type accepts the following properties:

Property	Default value	Description
path	/opt/opschain/<resource name>	The directory to check the repository out into. A relative value is resolved against /opt/opschain — the opschain user's home directory, where your project repository is checked out — so path 'app-config' checks out into /opt/opschain/app-config. Supply an absolute path to check out elsewhere. A value containing .. is rejected, as is a resource name containing / or .. when the default is used.
branch		The branch to check out. One of branch, tag or commit is required, and only one may be supplied.
tag		The tag to check out. An annotated tag is checked out at the commit it points to. A tag is checked out with a detached HEAD.
commit		The commit to check out, as a full 40-character SHA — an abbreviated SHA is rejected. A commit is checked out with a detached HEAD.
clone_during_dry_run	false	Whether to check the repository out when OpsChain loads your actions to discover them rather than to run a step — see when the repository is checked out. Set this to true if the repository holds YAML or similar configuration that your actions need in order to build the actions or the step tree.
git_remote	The resource name	The name of the project Git remote to check out. Set it to check out one Git remote more than once - see checking out one Git remote more than once. The remote must be available to the step under the git_remote.mountable setting. Cannot be combined with url.
url		The URL of a repository that is not one of the project's Git remotes. See checking out a repository that is not configured in the project. Supplying this means OpsChain fetches from the real remote rather than from a mirror, so any credentials the repository needs must be supplied too.
user		Username to authenticate an HTTPS url with. Not needed for a public repository.
password		Password or personal access token to authenticate an HTTPS url with.
ssh_key		The contents of the private key to authenticate an SSH url with — not a path to a key file. Store the key in an OpsChain property and read it back, rather than committing it to your Git repository.
passphrase		The passphrase for ssh_key, if it has one.
push_branch	The checked out branch	The branch the push action pushes to. Defaults to the checked out branch; required to push a tag or commit checkout. A branch other than the checked out one is created if it does not exist, and otherwise has the checkout's changes added to it - see pushing to another branch.
create_branch_from		A branch, tag or commit to create branch from when branch does not exist yet - see creating a branch. Only valid with branch, and only for one of the project's Git remotes.
commit_user_name		The author name for commits made in the checkout. Set it, with commit_user_email, before committing in the checkout: OpsChain configures no commit identity of its own, and commit_to_repo refuses to commit without both.
commit_user_email		The author email address for commits made in the checkout.

git_clone 'app-config' do
  branch :main
  clone_during_dry_run true
end

Pinning to a tag or a commit

Supply tag or commit instead of branch to check out a fixed point in the repository's history:

git_clone 'app-config' do
  tag 'v1.2.3'
end

git_clone 'app-scripts' do
  commit '3f786850e387550fdab836ed7e6dc881de23001b'
end

The default directory sits inside your project repository

/opt/opschain is where OpsChain checks your project's Git repository out, so the default target directory — /opt/opschain/<resource name> — is a directory inside that checkout. If your project repository already contains a directory of that name, OpsChain refuses to check out over it:

"/opt/opschain/app-config" already exists and is not a git working tree; refusing to check out over it

Name the resource something your project repository does not already contain, or set path to a location outside /opt/opschain.

Checking out one Git remote more than once

Set git_remote to check out the same project Git remote at several branches, tags or commits. Give each resource its own name, and name the remote in git_remote:

git_clone :config_main do
  git_remote 'app-config'
  branch 'main'
end

git_clone :config_release do
  git_remote 'app-config'
  tag 'v1.2.3'
end

Each resource checks out into its own directory - /opt/opschain/<resource name> by default, so /opt/opschain/config_main and /opt/opschain/config_release here - or into its own path. Both checkouts read from OpsChain's one mirror of app-config, so the second costs no extra network fetch.

Refer to each checkout by its resource name, not the remote's name: push_git_clone :config_main, commit_to_repo :config_release and git_clone_path(:config_main).

Two resources that push to the same branch behave like any two pushers. When the branch is one the second resource checked out, the second push is rejected once the first has moved it, as described in pushing changes. When it is a push_branch the second resource did not check out, its changes are added on top of the first push's - see pushing to another branch. Two resources may not share a directory, even when they check out the same remote - see idempotent checkouts.

If no Git remote of that name is available to the step, the step fails with No git remote named "app-config" (from the "git_remote" property of the "config_main" git_clone) is available to this step, followed by the names of the Git remotes that are available.

When the repository is checked out

OpsChain loads your actions for two different reasons, and clone_during_dry_run decides what happens in the second of them:

• To run a step. The repository is always checked out, whatever clone_during_dry_run is set to.
• To discover what your actions are, without running any of them. The repository is checked out only if clone_during_dry_run is true.

OpsChain loads your actions purely to discover them when it generates an asset template's actions, and when it works out the step tree of a change that runs from one of the project's Git remotes — the step tree of such a change is not known until its first step loads the actions. Running opschain-action --tasks (or any of the other task listing options) loads them the same way.

So if your actions read a checked out file while they are being loaded — to build the actions or the step tree from it, as in the pattern below — you need clone_during_dry_run true. Without it, discovery fails with a missing file error even though the step itself would have run correctly.

Declare the resource before the code that reads it

Because the clone happens as OpsChain evaluates the git_clone block, only code appearing after the block can read the cloned files. Reading them earlier in the file fails with a missing file error, which looks as though the clone never happened rather than as though it had not happened yet.

We suggest declaring your git_clone resources at the top of your actions, ahead of anything that depends on their contents:

git_clone 'app-config' do
  branch :main
  clone_during_dry_run true
end

config = YAML.load_file('/opt/opschain/app-config/config.yaml')

config['environments'].each do |environment|
  action "deploy_#{environment['name']}" do
    # ...
  end
end

Sharing repository declarations across several templates in one repository

A single Git repository often holds the actions for several asset templates, each in a folder named after its template code - so each template has its own actions.rb entry point. When several of them need the same repositories checked out, declare those once in a shared file and require it from each entry point:

git_clones.rb
web-server/actions.rb
database/actions.rb

# git_clones.rb
git_clone 'app-config' do
  branch :main
  clone_during_dry_run true
end

# web-server/actions.rb
require_relative '../git_clones'

config = YAML.load_file('/opt/opschain/app-config/config.yaml')

Declaring them in one place also removes a failure mode. Two templates that each declared app-config with a different branch, both checking out into the same directory, would raise the duplicate target error described below. With a single declaration they cannot disagree.

The ordering rule above still applies, one level further out: the require_relative must come before any code that reads the checked out files.

Set clone_during_dry_run on its own, not through properties

Setting it through the bulk properties form has no effect during a dry run, and fails silently:

git_clone 'app-config' do
  properties(clone_during_dry_run: true) # does nothing during a dry run
end

OpsChain skips bulk property assignment entirely while loading your actions for a dry run, so the property is never set, the repository is not cloned, and nothing reports a problem - it simply looks as though the setting is being ignored. Set it as its own property instead:

git_clone 'app-config' do
  clone_during_dry_run true
end

This applies to every resource property rather than only this one, but it matters most here, because this is the property whose whole purpose is to change what happens during a dry run.

Pushing changes

Call push_git_clone with the resource's name to push the commits made in the checkout back to the remote. The runner never holds the remote's credentials: OpsChain pushes on its behalf, and then refreshes its mirror so a later git_clone of the branch sees the new commits straight away. If that refresh fails, the push still succeeds and OpsChain records a warn:git_remote:push_refresh_failed event - the mirror catches up on its next refresh.

git_clone :my_repo do
  branch 'main'
  commit_user_name 'Release bot'
  commit_user_email 'release-bot@example.com'
end

action :bump_version do
  sh "sed -i s/1.0/1.1/ /opt/opschain/my_repo/VERSION"
  sh "git -C /opt/opschain/my_repo commit -am 'Bump version'"
  push_git_clone :my_repo
end

push_git_clone :my_repo runs the resource's push action, my_repo:push, which every git_clone of one of the project's Git remotes has. You can also list that action as a prerequisite, after the prerequisite that makes the commits - but not as a child step, as a child step starts from a fresh checkout and so has nothing to push.

Keep these rules in mind:

• Push in the same action as the commit. The checkout is reset to the mirror's tip whenever the resource is declared again - which happens for every step - so commits that have not been pushed by the end of the action are lost, even in a change that runs all its steps in one pod.
• The checked out branch only fast-forwards. Pushing to the checked out branch succeeds only if the branch has not moved on the remote since it was checked out, and only if the new commits descend from the commit that was checked out. A push that would overwrite someone else's commits fails with an error saying it was rejected - nothing is ever force pushed. Pushing to a different push_branch follows the rules in pushing to another branch.
• Push as often as you like. An action can push more than once, and can call commit_to_repo more than once: each push sends only the commits made since the previous one.
• Only for the project's Git remotes. A git_clone with a url already has the credentials it needs, so push raises an error rather than going through OpsChain - push with git directly instead.
• The artefact size limit applies. The new commits are sent to OpsChain as a Git bundle, which may not exceed the artefact size limit.
• Pushing needs the execute action on the project's /git_remotes path, as the user who created the change, started the agent or triggered the action generation doing the push.

Running push with no new commits does nothing to a branch that already exists, so running the push action on its own is harmless. When push_branch names another branch that does not exist yet, though, it creates that branch at the checked out commit.

Pushing to another branch

When push_branch names a branch other than the checked out one - always the case for a tag or commit checkout - what the push does depends on that branch:

• It does not exist. The push creates it, with the checkout's commits on top of the checked out commit.
• The checkout's commits already build on it. When the branch's latest commit is an ancestor of the checkout's - the checked out commit, say, or one of its ancestors - the push fast-forwards the branch to the checkout's commits, unchanged.
• It has commits the checkout does not. The push adds the checkout's commits on top of the branch's own, one new commit for each of them. Only the files a commit changed - added, modified or deleted - relative to the commit before it are applied; every other file on the branch is left as the branch has it. So what lands on the branch is the checkout's changes, compared with the commit that was checked out, not a copy of the whole checkout.

For example, to check out a release tag, generate a deployment manifest from it and record the manifest on a long-lived deployments branch:

git_clone :release do
  git_remote 'app-config'
  tag 'v1.2.3'
  push_branch 'deployments'
  commit_user_name 'Release bot'
  commit_user_email 'release-bot@example.com'
end

action :record_deployment do
  commit_to_repo :release, message: 'Deploy v1.2.3' do |repo_path|
    File.write(repo_path.join('deployed/manifest.yaml'), manifest.to_yaml)
  end
end

Only deployed/manifest.yaml is written to deployments; the rest of that branch - including any file v1.2.3 has a different version of - is untouched. A file the checkout changes is written as the checkout has it, though, replacing whatever the branch held at that path.

Keep these points in mind:

• The branch keeps its history. Nothing is force pushed: the new commits are added after the branch's latest commit. Each keeps its original author and message. Its committer is the resource's commit_user_name and commit_user_email when both are set, or else the original committer and date.
• Merge commits are flattened. A merge commit in the checkout becomes one ordinary commit, carrying the changes the merge made relative to its first parent. The commits merged in are not pushed separately.
• A clashing path fails the push. If a change cannot be applied because a path is a file on one side and a directory on the other, the push fails with an error such as Cannot apply the changes to deployments: 'config/app.yaml' appears as both a file and as a directory, and the branch is not changed.
• A branch that moves during the push is retried. If another pusher moves the branch while OpsChain is pushing to it, the push is attempted again against the branch's new latest commit, up to three times in all, before failing with the error saying it was rejected.

Committing changes with commit_to_repo

commit_to_repo runs a block against the checkout, commits whatever the block changed, and then pushes the commit. It saves you from running git add, git commit and push_git_clone yourself:

git_clone :my_repo do
  branch 'main'
  commit_user_name 'Release bot'
  commit_user_email 'release-bot@example.com'
end

action :add_file_to_repo do
  commit_to_repo :my_repo do |repo_path|
    File.write(repo_path.join('new_file.txt'), "here's some content")
  end
end

The block is given the checkout directory as a Pathname. OpsChain takes a snapshot of the directory before the block runs and another after it, and commits the files the block changed - new, modified and deleted files alike, including files Git does not track yet. A file the block changes back to its committed content has nothing to commit, so it is left out. Files Git ignores - through the repository's .gitignore, its .git/info/exclude or a global exclude file - are never committed unless they have been added with git add --force.

Option	Default	Description
push_changes	true	Whether to push the commit once it is made. Set it to false to commit without pushing - to push several commits at once with push_git_clone, for example.
message		The commit message. Defaults to Changes made by OpsChain change <change id> action "<action>".
paths		A glob, or a list of globs, limiting which changed files are committed - see choosing what to commit.

action :update_config do
  commit_to_repo :my_repo, push_changes: false, message: 'Update the configuration' do |repo_path|
    File.write(repo_path.join('config.yaml'), new_config.to_yaml)
  end
  commit_to_repo :my_repo, message: 'Update the version' do |repo_path|
    File.write(repo_path.join('VERSION'), '1.1')
  end
end

Choosing what to commit

Call commit_to_repo without a block to commit the changes already made in the checkout. It commits everything that differs from the checked out commit - new, modified and deleted files, and anything already staged with git add - except files Git ignores:

action :add_file_to_repo do
  File.write(git_clone_path(:my_repo).join('new_file.txt'), "here's some content")
  commit_to_repo :my_repo, message: "Here's a commit message", paths: '{new_file,other_file}.txt'
end

git_clone_path(:my_repo) returns the resource's checkout directory as a Pathname - its path, or /opt/opschain/<resource name> by default - so an action working on several repositories can find each checkout by its resource name. It can be called anywhere after the git_clone declaration, including while your actions are being loaded, but during a dry run without clone_during_dry_run the directory it returns has not been checked out.

Supply paths to commit only some of the changes, in either form. Each glob is matched against the changed file's path relative to the checkout, as Ruby's File.fnmatch matches it: * does not match across a /, **/ matches any number of directories, and {a,b} matches either alternative. Dotfiles and dot directories are matched like any other name, as in Git's own pathspecs, so paths: '**/*.yml' includes .github/workflows/ci.yml. A change no glob matches is left uncommitted. For example, paths: '*.txt' commits notes.txt but not docs/notes.txt, which needs paths: '**/*.txt'.

Keep these rules in mind:

• A commit identity is required. commit_to_repo raises an error before running the block unless the resource sets both commit_user_name and commit_user_email.
• Only the block's changes are committed. Anything that was already uncommitted in the checkout before the block ran is left uncommitted, unless the block changes that same file too - then the file is committed as the block left it, earlier edits included.
• Nothing changed, nothing happens. When there is nothing to commit - the block changed nothing, or paths matched none of the changes - OpsChain makes no commit and does not push.
• A failing block commits nothing. If the block raises an error, OpsChain neither commits nor pushes, and the step fails with the block's error.
• The rules for pushing changes still apply. In particular, a git_clone with a url cannot be pushed through OpsChain, so use push_changes: false for one and push with git yourself.
• Dry runs. When OpsChain loads your actions to discover them and the repository was not checked out, nothing is committed and the block is not run.

Creating a branch

Set create_branch_from to have OpsChain create branch on the remote when it does not exist yet, then check it out. The value can be a branch, a tag or a full commit SHA, and is looked up in OpsChain's mirror of the remote. When branch already exists, create_branch_from is ignored and the existing branch is checked out:

git_clone :my_repo do
  branch 'release/1.1'
  create_branch_from 'main'
end

The branch is created as the resource is declared, so it is created during actions discovery too when clone_during_dry_run is true - the only way an asset template's actions generation can create a branch, as no action runs then. If another step creates the same branch at the same time, the checkout uses the branch that step created.

Checking out a repository that is not configured in the project

Supply a url instead of naming one of the project's Git remotes. url and git_remote cannot both be set. There is no mirror for such a repository, so OpsChain fetches from the real remote over the network and you must supply whatever credentials it needs. Keep those in OpsChain properties rather than in your Git repository:

# a public repository needs no credentials
git_clone 'opschain-docs' do
  url 'https://github.com/LimePoint/opschain-docs.git'
  branch :master
end

# a private repository over HTTPS
git_clone 'app-config' do
  url 'https://github.com/my-org/app-config.git'
  user OpsChain.properties.github.user
  password OpsChain.properties.github.token
  branch :main
end

# a private repository over SSH - ssh_key takes the key itself, not a path to it
git_clone 'app-config' do
  url 'git@github.com:my-org/app-config.git'
  ssh_key OpsChain.properties.deploy_key
  passphrase OpsChain.properties.deploy_key_passphrase # omit if the key has no passphrase
  branch :main
end

note

Read the key out of a property directly, as above, rather than reading a file property from disk. OpsChain does not write file properties out when it loads your actions to discover them, so a File.read of a key file raises Errno::ENOENT in exactly the case clone_during_dry_run true exists to support.

A tag is looked up on the remote each time, so a tag that is moved on the remote is picked up by the next checkout. A commit is fetched from the remote by its SHA. Most Git servers allow that for any commit a branch or tag contains, but whether a commit that no branch or tag contains can be fetched depends on the server's configuration. When the server refuses, the step fails with Commit "<sha>" could not be fetched from the "<resource name>" git repository followed by Git's error.

Credentials are never passed on a command line, and a url that embeds credentials (https://user:password@host/…) is rejected — use user and password instead.

A repository checked out from a url does not go through the project's Git remotes, so the git_remote.mountable setting does not apply to it. The git_remote.ssh settings do: a checkout over SSH is given the same connection timeout and keepalives as OpsChain's own fetches, so a Git server that stops responding fails the step rather than hanging it.

Availability

Which of the project's Git remotes OpsChain will mirror into a step is controlled by the git_remote.mountable setting. By default every one of the project's active Git remotes is available, so git_clone works without any extra configuration.

The mirrors themselves are exposed to the step read-only — git_clone checks out from them, and a push goes through OpsChain rather than writing to them.

OpsChain refreshes the mirrors of the available Git remotes immediately before creating the runner pod, then mounts them for the life of that pod. branch :main therefore resolves to whatever the mirror's tip of main was when the pod was created — a commit pushed to the remote while the change is running is not picked up, and neither is a Git remote added to the project after the pod started.

The same applies to tags and commits. A tag resolves to the commit the mirror's copy of that tag pointed to when the pod was created, so a tag that is moved on the remote is picked up by the next pod. A commit never moves, but it must be reachable from a branch or tag the mirror holds.

If that refresh fails, because the remote is unreachable for example, OpsChain records a warn:git_remote:mountable_fetch_failed event and carries on rather than failing the change. The mirror the step checks out from may therefore be out of date, or missing altogether if the remote has never been fetched successfully.

Idempotent checkouts

Checking out a repository is idempotent. OpsChain leaves the target directory untouched when it already holds a checkout at the commit the requested branch, tag or commit currently resolves to, and in the same form: on the requested branch for a branch, or detached for a tag or commit. Otherwise OpsChain checks it out again — when the branch or tag has moved on, when a previous checkout was interrupted partway through, or when an earlier step checked the same commit out by a different branch, tag or commit.

note

Two git_clone resources in the same step that target the same directory with different configuration (for example, a different branch, tag, commit, Git remote or url) raise a clear error, rather than one silently overwriting the other. Two resources may check out the same Git remote as long as they target different directories.

Current limitations

• Running on a multi-node Kubernetes cluster is not yet supported for this feature.

OpsChain Kubernetes

Requiring opschain-kubernetes provides several resources for working with Kubernetes. These resources wrap the kubectl binary to allow you to perform some common Kubernetes operations.

Prerequisites

The kubectl binary must be available in your runner environment and is not included by default. To install kubectl, a custom Dockerfile must be included in your project's .opschain directory.

Below is an example Dockerfile RUN directive for adding kubectl to your runner.

...
# Run any Dockerfile commands that don't rely on the contents of the Git repository here to avoid rerunning them when the Git repo changes.
RUN curl -L -o /usr/local/bin/kubectl "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    chmod +x /usr/local/bin/kubectl
...

Authentication

There are multiple options available to authenticate with the Kubernetes cluster that you want to manage.

In-cluster service account config

By default, the opschain-kubernetes resource will use the opschain-runner service account to manage Kubernetes resources in the same cluster that OpsChain runs. You will need to grant the opschain-runner additional permissions to manage resources in your desired namespace(s) via additional RoleBindings or ClusterRoleBindings. Managing roles & permissions in your cluster is outside the scope of this documentation. Please see the Kubenetes RBAC documentation for more details.

Kubeconfig via OpsChain file properties

If you need to manage Kubernetes resources in another cluster, or don't want to use the opschain-runner service account as your identity, you can provide a custom kubeconfig file that will be read by kubectl. To do this, add a kubeconfig file via OpsChain file properties with the path /opt/opschain/.kube/config. See the OpsChain properties documentation for more information on adding file properties.

To use an alternative kubeconfig path set the KUBECONFIG environment variable via OpsChain properties.

Resource types

kubernetes_resource

The kubernetes_resource type provides a generic type with apply and delete actions for managing any valid Kubernetes resources via manifest files present in your project repository.

kubernetes_resource :nginx do
  manifest_path 'k8s/nginx.yaml'
  namespace 'myapp'
end
# provides nginx:apply and nginx:delete actions

kubernetes_daemonset, kubernetes_deployment, kubernetes_statefulset

The kubernetes_daemonset, kubernetes_deployment, and kubernetes_statefulset resource types provide actions for performing restart, scale, and wait operations on the standard 'workload' resources running within a Kubernetes cluster.

All three of these resource types provide the same functionality, but are provided as separately named types to account for how the resources are addressed within Kubernetes.

kubernetes_deployment :nginx do
  name 'nginx'
  namespace 'myapp'
  replicas 1
  wait_for_condition 'Available'
end
# provides nginx:restart, nginx:scale, and nginx:wait actions

Utilities

Logs

The kubernetes_daemonset, kubernetes_deployment, and kubernetes_statefulset types also provide access to a logs method on their controller.

The logs method requires you to pass a tail: <number of lines> argument to specify the number of log lines you would like returned. If you would like to return all log lines for the lifespan of the pod, you can use tail: -1. PLEASE NOTE that if your workload is a particularly noisy logger, this may result in a large amount of logs being buffered into memory, so use this with caution.

kubernetes_deployment :nginx do
  name 'nginx'
  namespace 'myapp'

  desc 'Wait until nginx deployment is available and show logs'
  action logs: ['nginx:wait'] do
    controller.logs(tail: 100).each do |line|
      OpsChain.logger.info line
    end
  end
end

By default, logs will return the logs for all containers in a pod, but you can also provide a container: '<container name>' argument to only return logs from a single container from within the pod.

action :logs do
  logs = controller.logs(tail: 100, container: 'app')
  # do something with logs
end

OpsChain SSH key pair

Requiring opschain-ssh-key-pair provides the ssh_key_pair resource type.

Resource type properties - SSH key pair

The ssh_key_pair resource type accepts the following properties:

Property	Default value	Description
key_path	/opt/opschain/.ssh	The location to generate the SSH key pair. Note: the default path is the opschain user's default SSH path.
private_key	id_rsa	The file name of the private key to generate (if a DSA type key is generated, the private key file name will default to id_dsa).
public_key	id_rsa.pub	The file name of the public key to generate (if a DSA type key is generated, the public key file name will default to id_dsa.pub).
type	RSA	The type of key to generate. Valid values are: - RSA - DSA
bits	4096	Determines the strength of the key in bits as an integer.
store_in	:environment	The OpsChain properties to store the generated key pair. Valid values are: - :environment the key pair will be stored in the OpsChain environment properties - :project the key pair will be stored in the OpsChain project properties - nil the key pair will not be automatically stored in OpsChain properties (see notes on key storage in the actions section below)
passphrase		Optional passphrase to assign to the private key.

Actions - SSH key pair

The ssh_key_pair resource type provides the following actions:

Action	Description
create	Creates an SSH public/private key pair inside the key_path folder with the filenames specified by public_key/private_key and optionally stores the files in OpsChain properties. Note: If the private_key or public_key exists in the key_path, they will be overwritten.
create_if_missing	Validates that the private_key and public_key exists in the key_path. If either is missing, generates a new key pair and optionally stores the key pair in the OpsChain properties.

Notes on key storage

The SSH key pair will be generated inside the OpsChain step runner container. By default the key pair will be stored in the OpsChain environment properties, making them accessible to future changes run in this environment (and subsequent steps in the current change). If you wish to use the key pair in other environments within the project, set the store_in resource property to :project. The key pair will then be stored in the OpsChain project properties and available to all changes run in that project.

If you do not wish to store the key pair in the OpsChain properties, store_in can be set to nil. Please note: If you do not store the generated keys in OpsChain properties, they will cease to exist when the step runner container is removed. For this reason, ensure the step stores the keys (e.g. in Hashicorp Vault, as a Kubernetes secret, on another server, etc.) to allow them to be used in the future.

Examples

The OpsChain AWS Ansible, OpsChain Confluent and OpsChain WebLogic example projects all make use of the ssh_key_pair resource type to generate SSH key pairs for their respective target containers.

OpsChain Terraform

Requiring opschain-terraform provides the terraform_config resource type. The resource type will accept any of the RubyTerraform command arguments as properties, but will only pass those supported by the command when the action is invoked.

Please see the RubyTerraform module documentation for further information about the available actions and their parameters.

note

RubyTerraform supplies vars to Terraform on the command line via multiple -var parameters. OpsChain overrides this logic by placing the input variables in a var file and supplying this to Terraform via the -var-file parameter to avoid encountering any command line length issues.

Prerequisites

opschain-terraform does not include the Terraform binary. Customers wishing to use the resource type will need to install Terraform in their project's step runner. This can be done by using a custom step runner Dockerfile. An example of this can be found in the OpsChain Confluent example.

Automatic Terraform initialisation

The terraform_config resource type will automatically execute terraform init in the OpsChain runner prior to running any Terraform action.

Automatic state storage

The terraform_config resource type will automatically store the terraform.tfstate file in the environment properties after running any Terraform action. This ensures that the file is available to subsequent steps in your change.

note

If the state_out property of Terraform is used, the resource type does not automatically store the file. Please use the store_file! feature (after moving the file to the desired location) to store the file.

Command argument defaults

Default values will be supplied for the following RubyTerraform command arguments:

Argument	Default value	Description
auto_approve	true	Indicates that Terraform should not require interactive approval before applying a plan.
chdir	pwd	The root directory of your project Git repository within the OpsChain step runner.
input	false	Indicates that Terraform should not attempt to prompt for input, and instead expect all necessary values to be provided by either configuration files or the command line.

tip

Resources can override these values if required.

Terraform automation environment variable

The Terraform TF_IN_AUTOMATION environment variable is automatically configured when running terraform_config actions. This will indicate to Terraform that there is some wrapping application executing terraform and cause it to make adjustments to its output to de-emphasize specific commands to run next. For further information see controlling Terraform output in automation.

Examples

The OpsChain Terraform example project demonstrates how the OpsChain Terraform resource type can be used.

The OpsChain AWS Ansible example project demonstrates how the OpsChain Infrastructure and OpsChain Terraform resource types can be combined with Ansible to deploy an nginx host on AWS.

The OpsChain Confluent example project demonstrates how the OpsChain Infrastructure and OpsChain Terraform resource types can be used together.