what is IPSEC protocol?

Internet protocol security (IPsec) is a set of protocols that provides security for Internet Protocol. It can use cryptography to provide security. IPsec can be used for the setting up of virtual private networks (VPNs) in a secure manner. Also known as IP Security.

Script to get the Total volume, snapshot in AWS

#! /bin/bash

echo "Collecting volume info"

aws  ec2 describe-volumes --region us-east-1 --query 'Volumes[*].{Size:Size}' --output table | awk 'NR>5 {print $2}' > /tmp/ax

aws  ec2 describe-volumes --region us-east-1  --query 'Volumes[*].{Size:Size,SnapshotId:SnapshotId}' --output table | grep snap | awk '{print $2}'> /tmp/ay

sumvol=0

sumsnap=0

#Volumes sum

for i in `cat /tmp/ax`

do 

sumvol=`expr $sumvol + $i`

done

echo "Total Volumes Size :" $sumvol

#Snapshot sum

for j in `cat /tmp/ay`

         do

         sumsnap=`expr $sumsnap + $j`

         done

echo "Total Snapshot Size :" $sumsnap

what is meant by ntp and how to setup ntp on node using the cookbook

What is ntp?
Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks. In operation since before 1985, NTP is one of the oldest Internet protocols in current use.

Manual process:

How to install ntp in centos7?

 yum install ntpd
 systemctl enable ntpd
 systemctl start ntpd
 systemctl status ntpd

How to check ntpdate?
[root@default-centos-73 ~]# ntpdate
30 Jul 20:58:52 ntpdate[4662]: no servers can be used, exiting


How to check sourceip address of the ntpserver of the node? 
ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-204.44.81.142.s 92.27.75.51      2 u   49  128  377   80.299  111.250  20.678
+pool-72-80-171- 209.51.161.238   2 u   64  128  377   43.400   92.246  18.945
+nimitz.jeremych 142.66.101.13    2 u  123  128  377   46.324  100.206  15.133
*block.steinhoff 204.9.54.119     2 u  121  128  377   11.698   95.785  14.339

How to check logs?
just grep for ntp in /var/log/messages
This will tell you if/where ntp activity is being logged:
grep log /etc/ntp.conf


Chef process:

 

How to create users using the chef cookbook?

Manual process:
useradd Sai  (To create the new user)
id sai (This is to verify,whether the user has created or not).
(uid=1002(sai) gid=1002(sai) groups=1002(sai))


Chef process using Recipe in the users cookbook:

user 'kevin' do
  comment 'default user'
  home '/home/kevin'
  shell '/bin/bash'
  password 'kevin'
end

This should be mentioned in the Chef - Recipe (default.rb).


How to write a test condition for the users cookbook?

describe user('Sai') do
it { should exist }
its('uid') { should eq 1001 }
its('shell') { should eq '/bin/bash' }
end

where we need to place this piece of code?

users (cookbook) >   test (directory) >  smoke  (directory) >   default  (directory) > default_test.rb

What is the necessity of test condition (Inspec test)?(the test condition will checkin for every 30mins on to the node,to know the status of the node and make sure whether it is satisfying the test condition or not).

whenever anything is not working properly/it is overrided / it is been removed/deleted,the test condition will fail,whenever there is a failure of test condition automatically the Chef-recipe is again applied on to the nodes and fulfill the requirement of the users cookbook.

Inspec Test

chef generate cookbook apache2

Once the cookbook is created,we have to go for `test` directory,Inside the test directory we have the recipes and the smoke directories.

We have to go through recipes directory

apache2  >   test  >   recipes >  we have to create the file (default_test.rb)

Inside this default_test.rb,we have the

# # encoding: utf-8

# Inspec test for recipe apache2::default

# The Inspec reference, with examples and extensive documentation, can be
# found at https://docs.chef.io/inspec_reference.html
# test regular file
#describe file('/etc/httpd/httpd.conf') do
#it { should exist }
#it { should be_file }
#end
describe package 'httpd' do
it { should be_installed }
end

describe service 'httpd' do
it { should be_disabled }
it { should be_running }
end

describe port 80 do
it { should be_listening }
end

These are the test conditions,we have to write before writing the recipes(main) in the cookbook.

-----------------------------------------------------------------------------------------------------------------------

#
# Cookbook Name:: apache2
# Recipe:: default
#
# Copyright 2017, YOUR_COMPANY_NAME
#
# All rights reserved - Do Not Redistribute
#
package 'httpd'

service 'httpd' do
  action %i[enable start]
end

template '/var/www/html/index.html' do
  source 'index.html.erb'
end
-------------------------------------------------------------------------------------------------------------------------

How to test the Cookbook?(using kitchen and chef-client --local-mode)

You can verify the Cookbook,that means verifying the recipe(like,we have to make sure the ruby chef code was written properly with all the correct syntax).

We can use the Kitchen,which is also called as the test kitchen,to test the runlist attached to the node.
We can also locally analyse the cookbook,by running the chef-client in the local-mode.

How to setup a Test Kitchen?
first and foremost we have to install the Vagrant and Virtualbox

kitchen acts as a provisioner,
(by default it takes the vagrant, basic requirements to test the kitchen, that is we have to install the vagrant and vagrant depends on the virtual box).
so we need to install vagrant and virtual box.

coming to docker, once when you install virtual box ,automatically the docker instance will setup on the virtual box.

step1:Install virtual box
step2:Download Vagrant
(https://www.vagrantup.com/downloads.html)
step3:vagrant -v (to check the version of vagrant).
step4:vagrant init hashicorp/precise64 b)vagrant up
first command creates a file and the second starts it
step5:we can ssh in to that box also, using the:
vagrant ssh.
—————————————————————————
Examples for the kitchen:
http://kitchen.ci/docs/getting-started/creating-cookbook
1)chef exec kitchen list
2)chef exec kitchen int
3)chef exec kitchen login
(or)
kitchen init --driver=kitchen-vagrant
(it includes the .kitchen.yml,chefignore).
once the above command is done, we have to do kitchen list
then we get the list of machines.
then select the required machine, we can create or converge the machines, the following are the commands for that:
1)kitchen create default-ubuntu-1204
the machine was created.
(or)
2) kitchen converge default-ubuntu-1204
login commands:
kitchen login default-ubuntu-1204




Or we can execute the Kitchen commands in the Following manner:
General Approach
 Once,we finished our cookbook writing,we need to go under two steps:
1)rubocop  -a   .
(everyone has to follow the single standard that is meaning of rubocop -a) 
A cookbook will be modified by several ppl in the organsation,inorder to bring everyone's modification in to the single-standard format rubocop -a  .  will help us to maintain the standard.
we are making uniform among the code.
offenses  =  standard where ruby chef script should be followed).
2)foodcritic  .
(to check the syntax’s and functionality of the code)
http://www.foodcritic.io/#FC008
(it is mainly to correct the code )
to correct the cookbook to to the standard for the ruby developers.
Steps of execution:
kitchen test
kitchen converge
kicthen verify
kitchen login

Note:
Make sure you perform all the command in the cookbook directory you want to test.
and make sure  you have the .kitchen.yml file to test with the kitchen commands.

Security bug in Bind (named)

I learned about a security issue with Bind, currently our version is affected. I ran yum update bind and the bug/ exploit is resolved.

Info on the issue:

From AWS:   https://alas.aws.amazon.com/ALAS-2017-858.html

From RedHat:  https://bugzilla.redhat.com/show_bug.cgi?id=1466189

Cookbook name apache

#   cookbook name:: apache

#    Recipe::  default

#    Install   apache  package

if node[‘platform_family’]   ==  “rhel”

    package   =  “httpd”

elsif  node[‘platform_family’]   ==  “debian”

     package   =  “apache2”

end

package  ‘apache2’  do 

               package_name package

                action   :  install

end 

service  ‘apache2’  do

               service_name   ‘httpd’

                action   [:start,   :enable]

end

           

we have to validate our code:

  ruby  -c default.rb

we have to validate our code:

 ruby  -c default.rb

similarly with the foodcritic,inorder to make sure it is correct.

    Foodcritic   default.rb

We can write a now recipe called the motd.rb

We are going simply doing the file resource type.

   Resourcename   path

  File   ‘/etc/motd’  do         (the file that we are going to manage).

      Content   ‘Hostname  is  this:   ‘

       (then we are putting node attribute information here).

       (what we are doing here is the combining text with the variable,(node attribute is a variable).

   Inorder to do this,we have to do the variable interpolation.

If we are planning to use the variable interpolation,we cannot use the  ‘ (single quote).

(hostname = node[‘hostname’]

hostname = node attribute of the hostname).

File   ‘/etc/motd’  do         (the file that we are going to manage).

      Content   ‘Hostname  is  this:   hostname (if you put lets this,it will interpret with the line,inorder to the variable hostname

Inorder to do that, we have to change from  single quote to the double quote.

hostname =  node[‘hostname’]

File   ‘/etc/motd’  do         (the file that we are going to manage).

      Content   “Hostname  is  this:   #{hostname}”

       end

This will allow us for the evaluation of the hostname.

As the content in the motd file.
----------------------------------------------------------------------------
version change:

there is a minor change or the major change.

Version     ‘0.1.1’

                   ‘1.    ---à  this is the major change.

                   ‘1.0.0   --à that is the minor change.

The . Is the minor change,

And the third one is considered the patch.

0.1.1     -à   the major change is the first item here.

The second after the .  is the  minor change.(would increment it).

 And the third one is the,be considered the patch.

If it is so major,that is the incompatable,with our node code.we perhaps incompatable with the api change.

We may to consider to change from the one to two.

2.0.0

but if it is a minor change:

we are going to the  0.2.1 

it’s for the patch,we are not fixing the bug,it will actually improve our cookbook.

In this way,it can be stored under version under the chef-server and ofcourse our runlist can define which version.

We can also revert back,if you don’t need to.

  

How to update ECS-AGENT version through chef cookbook

• We are installing ECS-AGENT on Docker hosts using  chef Role.
• We have three different environments  for Docker hosts (Dev,staging and production).
• W.r.t to three different enviroments,we have three different roles (ecs,ecs-staging,ecs-production).
• According to the chef architecture,the ecs-agent version should be updated automatically,because chef will checkin for every 30mins.

       Incase,ecs-agent has to be updated,we have to do following steps:
  

1. we have to remove  'ecs' role from the Runlist and run   `chef-client`  on the node.
2. Once the step1 is finished,we need to stop the 'ecs-agent` container and remove it.
3. Add the `ecs role` to the Runlist and run the chef-client on the node.
4. we can see the  `ecs-agent` running on the node.
5. we can verify the ecs-agent version by using this command  'curl http://localhost:51678/v1/metadata `
   
   

How to create a new user in the linux machine (centos)

Steps to Create a New Sudo User

1. Log in to your server as the root user. ssh root@server_ip_address.
2. Use the adduser command to add a new user to your system. Be sure to replace username with the user that you want to create. ...
3. Use the usermod command to add the user to the wheel group.
4. Test sudo access on new user account.

if you are signed in as the root user, you can create a new user at any time by typing:

adduser username

Note

After creating the user, for example you create ec2-user.

when you switch the user from root to the ec2-user,we will be using the sudo before any command to execute,Inorder to avoid the password asking part, you need to make some changes in the visudo file 

go to the root user   >  visudo    >   

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)

#includedir /etc/sudoers.d

kaushik   ALL=(ALL)       NOPASSWD: ALL

ec2-user    ALL=(ALL)       NOPASSWD: ALL

so,whenever you execute any command  in the above specified two users, it won’t ask you for the password.

If you are signed in as a non-root user who has been given sudo privileges, as demonstrated in the next section of this tutorial, you can add a new user by typing:

sudo adduser username

Next, you'll need to give your user a password so that they can log in. To do so, use the passwd command:

passwd username

We can do this by adding the user to the wheel group (which gives sudo access to all of its members by default) through the gpasswd command. This is the safest and easiest way to manage sudo user rights.

If you are currently signed in as the root user, type:

gpasswd -a username wheel

If you are signed in using a non-root user with sudo privileges, type this instead:

sudo gpasswd -a username wheel

Note

sudo lid -g wheel

The output will show you the usernames and UIDs that are associated with the group. This is a good way of confirming that your previous commands were successful, and that the user has the privileges that they need.

Deleting Users

If you want to delete the user without deleting any of their files, type this command as root:

userdel username

If you want to delete the user's home directory along with the user account itself, type this command as root:

userdel -r username

Example:

How to add a user to the existing group?

add a docker user to the docker group

sudo groupadd docker

sudo usermod -a -G docker  centos

List All Users In the System

cat /etc/passwd

to list all the users in the machine.


linux List users command

To list only usernames type the following awk command:

$ awk -F':' '{ print $1}' /etc/passwd

root
daemon
bin
sys
sync
games
man

References : https://www.cyberciti.biz/faq/linux-list-users-command/

Stop-current-running-crontab-tasks

Crontab -e and delete or add a # before the line u dnt want to execute


https://askubuntu.com/questions/313033/how-can-i-see-stop-current-running-crontab-tasks

Habitat for containers

Packages

A package refers to a binary distribution for a given piece of Habitat software that contains a software library or application as well as any configuration information for that software.

https://www.habitat.sh/docs/concepts-packages/

Keys:

Habitat has strong cryptography built into both the build system and the supervisor. This means there are several different kinds of keys.

Origin Keys

 Every package in Habitat belongs to an origin, and is cryptographically signed with that origin's private key.

Origin key cryptography is asymmetric; it has a public key that you can distribute freely, and a private key that you should keep safe.

Supervisors, by default, will refuse to run packages for which they do not have the public key. They use this public key to verify the integrity of the Habitat package they download, before running it. Supervisors can be provided the public key by pointing them at a depot that has it, or by putting the key on disk outside of Habitat.

https://www.habitat.sh/docs/concepts-keys/

Studio

The Habitat Studio is a clean, self-contained, minimal environment in which you can develop, build, and package software that is free from any upstream operating system distribution. All tools and dependencies included in the Studio are installed through Habitat packages, thus preventing any unwanted dependencies from being used by your package.

Create Packages

Packages are the cryptographically-signed tarballs that are uploaded, downloaded, unpacked, and installed in Habitat. They are built from shell scripts known as plans, but may also include runtime hooks and service configuration files that describe the behavior and configuration of a running package.

Create plans

At the center of Habitat packaging is the plan. This is a directory comprised of shell scripts and optional configuration files that define how you download, configure, make, install, and manage the lifecycle of the software in the package.

https://www.habitat.sh/docs/create-plans/

Build packages

Habitat packages are cryptographically-signed tarballs with a .hart extension that are built from plans. You can build a package in two ways: interactively from inside a studio, and non-interactively.

https://www.habitat.sh/docs/create-packages-build/

Container orchestration with Habitat

Habitat packages may be exported with the supervisor directly into a Docker or ACI-formatted container, but frequently the container itself will run within a container orchestrator such as Kubernetes or Mesos. Container orchestrators provide scheduling and resource allocation, ensuring workloads are running and available. Containerized Habitat packages may run within these runtimes, managing the applications while the runtimes handle the environment surrounding the application (ie. compute, networking, security).

https://www.habitat.sh/docs/container-orchestration/

Build Docker containers from scratch with habitat

Steps	Command/Description
Installation	https://www.habitat.sh/docs/get-habitat/
CLI Configuration	hab setup (configure the CLI).
Start with a plan	hab plan init ruby
Inside the ruby directory	$ ls -la ruby / config/ hooks/ default.toml plan.sh
plan.sh	plan.sh is a bash script that describes how to build your package.
Enter the studio	hab studio enter (build and test your Habitat packages).
Build the package	build (Within your studio environment, run).
Export the package as a Docker image	hab pkg export docker (nameoftheorigin)/ruby
Testing the build	docker run --rm --entrypoint=/bin/bash nameoftheorigin/ruby  -c irb

Reference:

https://opensolitude.com/2017/03/08/build-docker-containers-from-scratch-with-habitat.html

chef habitat plan syntax

pkg_name

Required. Sets the name of the package. This will be used along with pkg_origin, and pkg_version to define the fully-qualified package name, which determines where the package is installed to on disk, how it is referred to in package metadata, and so on. A pkg_name can contain upper and lowercase letters, numbers, dashes, and underscores. 

Example :  pkg_name=zlib

pkg_origin

Required unless overridden by the HAB_ORIGIN environment variable. The origin is used to denote a particular upstream of a package. A pkg_origin can contain upper and lowercase letters, numbers, dashes, and underscores. 

Example : pkg_origin=Habitat

pkg_shasum

Required if a valid URL is provided for pkg_source or unless do_verify() is overridden. The value for pkg_shasum is a sha-256 sum of the downloaded pkg_source. If you do not have the checksum, you can easily generate it by downloading the source and using the sha256sum or gsha256sum tools. Also, if you do not have do_verify() overridden, and you do not have the correct sha-256 sum, then the expected value will be shown in the build output of your package. 

pkg_shasum=36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d

pkg_description

Required for core plans, optional otherwise. A short description of the package. It can be a simple string, or you can create a multi-line description using markdown to provide a rich description of your package. 

example :

  pkg_description=$(cat << EOF

  # My package description

  This is the package for the foo library. It's pretty awesome.

  EOF

  )

Optionals:

pkg_version

pkg_source

pkg_deps

pkg_lib_dirs

pkg_include_dirs

pkg_bin_dirs

pkg_pconfig_dirs

pkg_svc_run

pkg_exports

pkg_exposes

pkg_binds

pkg_binds_optional

pkg_interpreters

pkg_svc_user

pkg_svc_group

pkg_upstream_url

pkg_license

pkg_filename

pkg_build_deps

Variables:

The following variables can be used in your plans to help get binaries and libraries to build and install in the correct locations in your package.

$pkg_prefix

The absolute path for your package.

$CFLAGS

C compiler options.

$PREFIX

Where to install the software; same as $pkg_prefix

Callbacks 

When defining your plan, you have the flexibility to override the default behavior of Habitat in each part of the package building stage through a series of callbacks. To define a callback, simply create a shell function of the same name in your plan.sh file and then write your script. If you do not want to use the default callback behavior, you must override the callback and return 0 in the function definition.

These callbacks are listed in the order that they are called by the package build script.

do_build()

The default implementation is to update the prefix path for the configure script to use $pkg_prefix and then run make to compile the downloaded source. This means the script in the default implementation does ./configure --prefix=$pkg_prefix && make. You should override this behavior if you have additional configuration changes to make or other software to build and install as part of building your package.

do_check()

The default implementation runs nothing during post-compile. An example of a command you might use in this callback is make test. To use this callback, two conditions must be true. A) do_check() function has been declared, B) DO_CHECK environment variable exists and set to true, env DO_CHECK=true.

do_install()

The default implementation is to run make install on the source files and place the compiled binaries or libraries in HAB_CACHE_SRC_PATH/$pkg_dirname, which resolves to a path like /hab/cache/src/packagename-version/. It uses this location because of do_build() using the --prefix option when calling the configure script. You should override this behavior if you need to perform custom installation steps, such as copying files from HAB_CACHE_SRC_PATH to specific directories in your package, or installing pre-built binaries into your package.

Hooks

Each plan can have a hooks subdirectory that specifies any of the hooks or asynchronous callbacks described in this section. Each hook is a script with a shebang defined at the top to specify the interpreter to be used.

References : 

https://www.habitat.sh/docs/reference/plan-syntax/#variables

Patching and rebooting the servers

How to reboot the routers?
ipsec pluto - IPsec IKE keying daemon
ipsec whack - control interface for IPSEC keying daemon
pluto is the router

commands:

route -n

yum check-update

chkconfig  --list  | grep -i ipsec 

(it will check and let us know, the process will be up at the boot time).

Always the run level 3 should be ON.

Patching

yum update -y (to update all the packages)

Note: If the run level 3 is not ON  for the ipsec,we need to add them.

chkconfig --level 3 ipsec on

ftp server

yum update

chkconfig  --list | grep ftp

run level 3 should be on (it means it has to be start on boot time).

Note: If the run level 3 is not ON  for the ipsec,we need to add them.

chkconfig --level 3 ftp on

Node applications

yum update -y

ps aux | grep node 

pm2 status www

reboot 

pm2 start www

Note:need to understand how to add this at the boot time.

Bamboo server

yum update

ps aux | grep bam

reboot 

find / ~name  start-bamboo.sh

(visit bamboo site for more information).

./start-bamboo.sh

Path : (cd /opt/bamboo/……………/start-bamboo.sh).

Note:need to understand how to add this at the boot time.

How to change the hostname of a linux system

http://www.ducea.com/2006/08/07/how-to-change-the-hostname-of-a-linux-system/

PM2 ---> P(rocess) M(anager)2

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.


How to use that ?
https://github.com/Unitech/pm2