Kaushik Gattu: September 2017

Thursday, September 14, 2017

A little help from the Community

Supermarket!

The hard way...

§ Up to this point we have been writing our own cookbooks.

§ While this allows us to write very specific cookbooks tailored to our

environment, its not very practical.

§ When Gap started with Chef, we had to write almost all of the

original base cookbooks including gapNetboot (use to provision nodes

built the gap way), gapUDS (for authentication), gapRepos, gapSudo,

etc.

§ For most of these applications, the community has already tackled

this problem for us...

The easy way...

http://supermarket.chef.io

§ We can leverage Chef's active community and use some of the

cookbooks that have already been written for mass consumption

§ Hundreds or cookbooks already exist for a large number of use cases

and purposes. Many (but only a fraction) are maintained by Chef.

§ Think of it like RubyGems.org, CPAN.org, or other focused plugin-style

distribution sites.

Like Prego, its in there...

http://supermarket.chef.io

Look for a cookbook that manages an application you are interested in:

§ apache? (apache2?)

§ tomcat?

§ mysql?

§ postgresql?

§ mongodb?

§ nodejs?

Yep. Its in there...

Problem: Manager wants our nodes to run

chef-client periodically

http://supermarket.chef.io

§ Up to this point, we've been running chef-client manually. This isn't

how we run it in any of our environments...

§ Problem: To keep out infrastructure in compliance, we need to run

the chef-client every 30 mins. We should also delete the validation

pem from our node, now that we don't need it anymore.

§ Solution: Use the community cookbook provided by Chef!

Search for chef-client

http://supermarket.chef.io

Always check the code!

https://github.com/opscode-cookbooks/chef-client

chef-client

http://supermarket.chef.io

Search the chef-client cookbook in the supermarket.chef.io

Use Berks to install this cookbook (and its

dependencies) to your Chef Server

§ Life before Berks involved a lot of downloading a cookbook and then

trying to upload it to the Chef Server only to see that it would fail

with a missing dependency.

§ If you chef the metadata.rb of the chef-client cookbook you will

notice that it has 3 dependencies.

§ BUT, those cookbooks may have their own dependencies...

§ In fact, the chef-client cookbook has a total of 4 dependent

cookbooks that need to be uploaded to the Chef Server before we can

finally try to upload chef-client.

§ Berks to the rescue...

Berksfile

(chef-repo/Berksfile)

source 'https://supermarket.chef.io'!

cookbook 'chef-client'!

berks install

Resolving cookbook dependencies...

Fetching cookbook index from

https://supermarket.chef.io...

Installing chef-client (4.3.1)

Installing chef_handler (1.2.0)

Installing cron (1.6.1)

Installing logrotate (1.9.2)

Installing windows (1.38.1)

berks upload

Uploaded chef-client (4.3.1) to: 'https://

centos6.novalocal:443/organizations/dfisher'

Uploaded chef_handler (1.2.0) to: 'https://

centos6.novalocal:443/organizations/dfisher'

Uploaded cron (1.6.1) to: 'https://centos6.novalocal:

443/organizations/dfisher'

Uploaded logrotate (1.9.2) to: 'https://

centos6.novalocal:443/organizations/dfisher'

Uploaded windows (1.38.1) to: 'https://

centos6.novalocal:443/organizations/dfisher'

Add chef-client and chefclient::

delete_validation to base.rb

(roles/base.rb)

name 'base'

description 'Base Server Role'

run_list 'recipe[motd]','recipe[repos]','recipe[chefclient]','

recipe[chef-client::delete_validation]'

Upload your role to the Chef Server

knife role from file base.rb

Updated Role base!

Run chef-client

sudo chef-client

[2015-08-19T06:57:05+00:00] INFO: Forking chef instance to converge...!

Starting Chef Client, version 12.4.1!

[2015-08-19T06:57:05+00:00] INFO: *** Chef 12.4.1 ***!

[2015-08-19T06:57:05+00:00] INFO: Chef-client pid: 31951!

[2015-08-19T06:57:07+00:00] INFO: Run List is [role[base], role[webserver]]!

[2015-08-19T06:57:07+00:00] INFO: Run List expands to [motd, repos, chefclient,

chef-client::delete_validation, mysite]!

[2015-08-19T06:57:07+00:00] INFO: Starting Chef Run for node1!

[2015-08-19T06:57:07+00:00] INFO: Running start handlers!

[2015-08-19T06:57:07+00:00] INFO: Start handlers complete.!

[2015-08-19T06:57:07+00:00] INFO: HTTP Request Returned 404 Not Found:!

resolving cookbooks for run list: ["motd", "repos", "chef-client", "chefclient::

delete_validation", "mysite"]!

[2015-08-19T06:57:08+00:00] INFO: Loading cookbooks [motd@0.1.0, pci@0.1.0,

repos@0.1.0, chef-client@4.3.1, cron@1.6.1, logrotate@1.9.2, windows@1.38.1,

chef_handler@1.2.0, mysite@0.1.0]!

Synchronizing Cookbooks:!

Chef the Chef-Client process

ps –ef | grep chef-client

gaptech 379 31762 0 07:32 pts/0 00:00:00 grep chef-client

root 32565 1 0 06:57 ? 00:00:00 /opt/chef/embedded/bin/ruby /

usr/bin/chef-client -d -c /etc/chef/client.rb -P /var/run/chef/client.pid -i

1800 -s 300

Gap runs its own internal Supermarket

http://supermarket.phx.gapinc.dev

Like your own Tomato Sauce, we control

what goes in there...

http://supermarket.chef.io

Look for a cookbook that manages an application you are interested in:

§ apache? (apache2?)

§ tomcat?

§ mysql?

§ postgresql?

§ mongodb?

§ nodejs?

Yep. Its in there...but much fewer versions.

(and less documentation)

How do I get my version of cookbook into

our internal Supermarket?

§ Like the Berksfile we just edited, we use a Master Cookbooklist

(which happens to look like a giant Berksfile)

§ We keep it in source control

§ http://github.gapinc.dev/enterprise-management/cookbooklist

§ Want to add to it?

§ Make a Pull Request and someone from Automation Team will review

it for you

§ Let's see how to do that

How do I get my version of cookbook into

our internal Supermarket?

using github repos.

Clone the Cookbooklist repo

git clone http://github.gapinc.dev/enterprise-management/cookbooklist.git

Cloning into 'cookbooklist'...

remote: Counting objects: 2259, done.

remote: Total 2259 (delta 0), reused 0 (delta 0), pack-reused 2259

Receiving objects: 100% (2259/2259), 329.34 KiB | 0 bytes/s, done.

Resolving deltas: 100% (676/676), done.Checking connectivity... done.

Make a Branch

git checkout -b JPC/TrainingTest

flexo:cookbooklist jcook$ git checkout -b JPC/TrainingTest!

Switched to a new branch 'JPC/TrainingTest'

Edit the Berksfile

$ vi Berksfile

source 'https://supermarket.chef.io'

group :community_cookbook do

cookbook 'testcookbook', '0.1.0'

cookbook 'apache2', '1.8.4'

cookbook 'apt', '2.7.0'

cookbook 'ark', '0.9.0'

cookbook 'automysqlbackup', '2.0.1'

cookbook 'aws', '1.0.0'

cookbook 'build-essential', '2.2.3'

cookbook 'chef-server', '2.0.0'

...

Commit the Change

$ vi Berksfile

[JC] Adding testcookbook

# Please enter the commit message for your changes. Lines starting!

# with '#' will be ignored, and an empty message aborts the commit.!

# Explicit paths specified without -i or -o; assuming --only paths...!

# On branch JPC/TrainingTest# Changes to be committed:!

# modified: Berksfile#!

Push your local branch to Github

git push -u origin JPC/TrainingTest

Username for 'http://github.gapinc.dev': jo1a8y1

Password for 'http://jo1a8y1@github.gapinc.dev':

Counting objects: 3, done.

Delta compression using up to 8 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (3/3), 339 bytes | 0 bytes/s, done.

Total 3 (delta 1), reused 0 (delta 0)

To http://github.gapinc.dev/enterprise-management/cookbooklist.git

* [new branch] JPC/TrainingTest -> JPC/TrainingTestBranch JPC/

TrainingTest set up to track remote branch JPC/TrainingTest from origin.

Create the Pull Request

http://github.gapinc.dev/enterprise-management/cookbooklist.git

Wait for someone from the Automation

team to approve the PR

The Automation team will look at your request and work with you to

figure out if we can approve it.

§ There could be a variety of reasons why they might not be able to

immediately approve your request, but they should work with you to

find a solution.

§ Normally, they will just approve it.

****Jenkins will pull the new Berksfile and

push your cookbook everywhere******

Knowledge Check

§ Why do we want to use Community cookbooks

§ What is the first thing you should read when downloading a cookbook?

§ Who vets cookbook in the Public Supermarket?

§ Who vets cookbooks in the Internal Supermarket?

chef-Environments for the People!

What is an Environment?

§ Environments

− Used to define the workflow our your

application

− Generally used to segment the

different life stages of your application

> Dev / Integration

> VDEV

> Staging

> Production

§ GID is configured this way

§ Corp is defined by Data Center, but

is the process of changing to use

best-practices

§ Environments also define policy

− Environments can also define or

override attributes

> Define different API locations for Dev /

Stage / Prod

> Dev sends alerts to team, Prod sends

alerts to L1

− Ideal place to pin to a specific version

of a cookbook

Chef Best Practice: Environment or Org?

§ We cannot share cookbooks between organizations

§ Organizations are like stand alone chef-servers

§ Best Practice: If you need to share cookbooks or roles, you likely want

an Environment rather than an organization.

§ Environments allow for isolating resources within a single organization

How Gap differs from this Best Practice

§ Currently at the Gap, organizations are defined by their old business

units:

− Gap (the old Corp)

− GID (the old GID)

− Stores

§ Environments are used differently between the organizations

− GID follows best practice and pins cookbook versions in the Environment

− Stores defines Environments for classes of stores and they also pin.

− Corp defines Environments as Data Centers.

> Corp cannot pin in its ENvironments.

> They depend on pinning to happen in a Role

> This is going to change...eventually.

Use knife to show available versions of

mysite

knife cookbook show mysite

mysite 0.2.0 0.1.0

Use knife to list current environments

knife environment list

_default

Create the Development Environment

(environments/dev.rb)

name 'dev'!

description 'For developers!'!

cookbook 'mysite', '= 0.2.0'

Cookbook Version Constraints

§ By default, use = (Equal to)

§ There are lots of other options, but in Environments equality '=' is the

recommended practice

§ Check out https://docs.chef.io/cookbook_versions.html for more

information about Version Constraints.

Upload our new dev environment

knife environment from file dev.rb

Updated Environment dev

Use knife show the dev environment

knife environment show dev

chef_type: environment

cookbook_versions: mysite: = 0.2.0

default_attributes:

description: For developers

json_class: Chef::Environment

override_attributes:

Move your node to the dev environment

(nodes/node1.json)

{

"name": "node1",

"chef_environment": "dev",

"run_list": [

"role[base]",

"role[webserver]"

"normal": {

"tags": [

]

}

Use knife show the dev environment

knife node from file node1.json

Updated Node node1!

Run chef-client

sudo chef-client

[2015-08-17T02:24:22+00:00] INFO: Chef Run complete in

3.48210582 seconds!

Running handlers:!

[2015-08-17T02:24:22+00:00] INFO: Running report handlers!

Running handlers complete!

[2015-08-17T02:24:22+00:00] INFO: Report handlers complete!

Chef Client finished, 0/15 resources updated in 5.507357093

seconds!

Use knife to see what cookbooks node1

used in its last Chef Run

knife node show node1 -a cookbooks

node1:

cookbooks:

motd:

version: 0.1.0

mysite:

version: 0.2.0

pci:

version: 0.1.0

repos:

version: 0.1.0

Create the prod environment

(environments/prod.json)

{

"name": "prod",

"description": "For Prods!",

"cookbook_versions": {

"mysite": "= 0.1.0"

"json_class": "Chef::Environment",

"chef_type": "environment",

"override_attributes": {

"pci": {

"in_scope": true

}

Best Practice: Environments get override

attributes

§ Environments are the one place where we can consistently use

overrides

§ Remember, we are overriding an attribute like:

− different API locations for Dev / Stage / Prod

− different alerting requirements for monitoring

− etc..

§ If you always set default attributes in your cookbooks and role, then

the override in Environments will win.

Use knife show the dev environment

knife environment from file prod.json

Updated Environment prod

Check to see what's in our /etc/motd

cat /etc/motd

node1 is property of Gap, Inc.!

Run chef-client

[

sudo chef-client

[2015-08-17T02:43:09+00:00] INFO: Run List is [role[base], role[webserver]]!

[2015-08-17T02:43:09+00:00] INFO: Run List expands to [motd, repos, mysite]!

[2015-08-17T02:43:09+00:00] INFO: Starting Chef Run for node1!

[2015-08-17T02:43:09+00:00] INFO: Running start handlers!

[2015-08-17T02:43:09+00:00] INFO: Start handlers complete.!

[2015-08-17T02:43:09+00:00] INFO: HTTP Request Returned 404 Not Found:!

resolving cookbooks for run list: ["motd", "repos", "mysite"]!

[2015-08-17T02:43:09+00:00] INFO: Loading cookbooks [motd@0.1.0, pci@0.1.0, repos@0.1.0,

mysite@0.1.0]!

[2015-08-17T02:43:09+00:00] INFO: Removing cookbooks/mysite/attributes/default.rb from the

cache; its is no longer in the cookbook manifest.!

[2015-08-17T02:43:09+00:00] INFO: Removing cookbooks/mysite/templates/default/

index.html.erb from the cache; its is no longer in the cookbook manifest.!

[2015-08-17T02:43:09+00:00] INFO: Removing cookbooks/mysite/templates/default/custom.erb

from the cache; its is no longer in the cookbook manifest.!

Check to see what's in our /etc/motd

cat /etc/motd

node1 is property of Gap, Inc.!

This server is in-scope for PCI compliance.

Use knife to see what cookbooks node1

used in its last Chef Run

knife node show node1 -a cookbooks

node1:

cookbooks:

motd:

version: 0.1.0

mysite:

version: 0.1.0

pci:

version: 0.1.0

repos:

version: 0.1.0

Why is this still there?

Welcome to Gap inc!!

We love athleta

172.16.232.5:8000

Rollbacks and Desired State Best Practice

§ Chef is not magic – it manages state for declared resources

§ We just rolled back to an earlier version of the mysite cookbook

§ While the recipe converged properly, looking at the node will show

that Apache is still configured as it was after running version 0.2.0 of

the cookbook

§ A better way to ensure a smooth rollback:

− Write contra-resources to clean up

− Put those new resources in a new version of the cookbook

- Or possibly, nuke the node and build a new one!

Chef always wants to move forward...

Wednesday, September 13, 2017

chef-Role Playing

Reducing Complexity through Abstraction

What are Roles?

How should we use them?

§ Up till now, we've just been adding recipes directly to our node's run

list. Across a large number of nodes, this can be time consuming,

error prone and highly inefficient.

§ Roles allow you to easily encapsulate all of the recipes and attributes

required for a server to become:

− A Web Server

− An Application Server

− A Database

§ In practice, Roles make it easy to configure many nodes identically

with the minimal amount of typing.

Best Practice: Roles belong in your source

control

§ At the Gap, roles live in the gapChef repo in the roles directory

§ This is the only way to create them at Gap.

§ You must check in your role to Git and then the Jenkins pipeline will

push your role to the Chef Servers

§ This is most important, because roles, like nodes and data bags are

unversioned objects in Chef.

§ Source control makes the tracking down of changes of these types of

files easy.

Create the webserver role

(roles/webserver.rb)

Create the webserver role

§ A Role has

− name

− description

− run_list

§ We can also set node attributes

with a role.

{

"name": "webserver",

"description": "Web Server",

"json_class": "Chef::Role",

"default_attributes": {

"mysite": {

"sites": {

"athleta": {

"port": 8000

}

"chef_type": "role",

"run_list": [

"recipe[mysite]"

]

}

This is JSON data, it takes " " and

loves commas.

Upload the webserver role

knife role from file webserver.json!

Updated Role webserver!!

Use knife to view role on Chef Server

knife role show webserver

chef_type: role

default_attributes:

mysite:

sites:

athleta:

port: 8000

description: Web Server

env_run_lists:

json_class: Chef::Role

override_attributes:

run_list: recipe[mysite]

knife search role 'run_list:recipe\[mysite\]'

1 items found

chef_type: role

default_attributes:

mysite:

sites:

athleta:

port: 8000

description: Web Server

env_run_lists:

json_class: Chef::Role

override_attributes:

run_list: recipe[mysite]

Best Practice: Nodes belong in your source

control

§ At the Gap, nodes also live in the gapChef repo..but in the nodes