script on Zwindler's Reflection

Your very first VM in Azure… deployed with Ansible, of course!

Tue, 14 Aug 2018 11:45:31 +0000

Haven’t you already wrote an article about Ansible and VM provisioning?

You might have read already a few articles on Ansible on this blog. I really like Ansible. I already wrote a few articles on this topic, even a few in english, and there are a few playbooks on my Github account if you want to check it out.

[French readers]Au cas où vous l’auriez manqué, la version française de cet article est disponible ici[/French readers]

One of the articles that you like the most, especially my english speaking viewers, is the one about deploying virtual machines on VMware vSphere platform. But now that I have moved on past VMware and started working on cloud providers (mostly the big ones, AWS, GCP and Azure), I felt it was time to share my automation playbooks about that also ;)

As you might have guessed, Ansible and cloud providers teams have been working hard to provide integrations. You can have a look at the BIG list of modules for cloud integrations only!
https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html

That’s especially true for Azure and Microsoft teams, which are relatively available and that provide regular updates (this is going to be the topic of another article, #teasing).

Prerequisites

Now that I introduced the subject, let’s deploy a VM on Azure. Yay!

Good news is that’s it’s almost the same thing as deploying a VM in vSphere. We have prerequisites, parameters to fill in and pouf, a VM appears.

Bad news is that prerequisites are going to be a little bit more tedious to install/configure, but worry not, I’ll give you all the steps ;)

There are two guides giving us info on how to configure our workstation to start working with Azure:

Install

On a mint/debian/ubuntu, get the necessary python modules :

apt install python-pip ansible
pip install setuptools wheel
pip install ansible[azure]

Configure

To deploy a VM in Azure, you need… an Azure account!

Thanks, captain obvious.

Ok, so assuming you don’t have one, you have to know that you can get a 200$ free credit. To open the account, simply go to this page

The only two issues with this is that this credit expires after 30 days (while AWS of GCP gives 365 days!) and that you need a credit card. I know that having to register a credit card can be a little frightening.

If you read all the informations, they claim that it’s only to avoid bots and that they won’t get money from it. If you consume all you credit or move past the 30 days, the account will be locked. Personnaly it worked great indeed and I didn’t lost a cent, but you shouldn’t believe me and be careful to avoid a bad surprise I case it dooesn’t work as well for you.

Once we have our account, we have to configure our workstation. The easiest way to do this is to use az cli for linux (but then you need to install it). You can also configure the files by hand (following the guides I gave previously).

az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CCHMTROLL to authenticate.

After that, you should have a .azure in your homedir

ll .azure
total 8
drwxrwxrwx 1 zwindler zwindler 512 Aug 4 17:00 ./
drwxr-xr-x 1 zwindler zwindler 512 Aug 4 16:56 ../
-rw------- 1 zwindler zwindler 7124 Aug 4 17:00 accessTokens.json
-rw-rw-rw- 1 zwindler zwindler 5 Aug 4 16:56 az.json
-rw-rw-rw- 1 zwindler zwindler 5 Aug 4 16:56 az.sess
-rw-rw-rw- 1 zwindler zwindler 333 Aug 4 17:00 azureProfile.json
-rw-rw-rw- 1 zwindler zwindler 66 Aug 4 17:00 clouds.config
-rw------- 1 zwindler zwindler 27 Aug 4 16:56 config

Soooo… we deploy this VM or not?

Yeah, now that our workstation is configured, this becomes easy again!

The Azure module to create a VM in a playbook is azure_rm_virtualmachine. Documentation can be found here docs.ansible.com/ansible/latest/collections/azure/azcollection/azure_rm_virtualmachine_module.html

Here’s one of the examples given in the documentation:

- name: Create a VM with managed disk
azure_rm_virtualmachine:
resource_group: Testing
name: testvm001
vm_size: Standard_D4
managed_disk_type: Standard_LRS
admin_username: adminUser
ssh_public_keys:
- path: /home/adminUser/.ssh/authorized_keys
key_data: < insert your ssh public key here... >
image:
offer: CoreOS
publisher: CoreOS
sku: Stable
version: latest

Most of the options given here are a strict minimum. There are many many more. I’ll explain a little what’s what:

resource_group
On azure, there are multiple logic entities to segment resources between each other. Subscription is the highest level of segmentation (mostly used for multitenancy, environment segregation or billing) and resources groups. You can have multiple subscription in a same account, and many resource groups in a single subscription.

name
The name of the resource we are going to deploy.

vm_size
Now we enter inside the topic. This is a size of the VM (in terms of vCPUs and GB of RAM) we are going to give our VM. A guide of the available size can be found here => docs.microsoft.com/fr-fr/azure/virtual-machines/windows/sizes-general

managed_disk_type

The type of the virtual disk we are going to create at the same time. Basically, will your disk be on full SSD disk array or not. You won’t find much information on exactly how your data is stored aside from SLAs (availability and performance).

For example, we know that a Premium disk of 128 GB will have a performance threshold of 500 IOPS and 100 MB/s. If you add just one GB (from 129 GB to 256), you’ll get 1100 IOPS and 125 Mo/s. Don’t expect to swindle Azure by have the performance of a 256 GB disk for the price of 128+1 GB, because a the moment you move past 128 GB, you’ll pay for a 256 GB (even if you ask only for 129, and even if you only you 10 GB).

admin_username & ssh_public_keys

Self explainatory.

image

On Azure, you can access preconfigured images:

If we want a Ubuntu 16.04:

image:
offer: UbuntuServer
publisher: Canonical
sku: '16.04.0-LTS'
version: latest

If we want a CentOS 7.4 :

image:
offer: CentOS
publisher: OpenLogic
sku: '7.4'
version: latest

Mooooar!

Ok, I gave you a little piece of YAML and I explained it to you. That’s nice, but you want more. Can we have more, you ask?

Of course, you can!

In fact, you need more to have a working VM. In Azure, as we are not in our own datacenter where everything has been previously set up from network to hypervisors, we need more objects (resources) than just a VM.

I already told you about it, we first need a resource group, in which we’ll put our VM. But, we also need a virtual network interface that we will attach to the VM. This vNIC will have to have a public IP address if we are going to connect on it. Also, we will need a virtual network for the vNIC, and maybe add some basic firewalling rules.

We could do all this by hand… :(

Or… we could automated it all with an Ansible playbook !!!

---
- name: "Create Azure resources"
connection: local
hosts: all
vars_prompt:
- name: "location"
prompt: "Choose region to deploy VMs"
private: no
default: "westeurope"
- name: "project_prefix"
prompt: "Choose a prefix for all the resources"
private: no
default: "test"
- name: "instances_number"
prompt: "Choose a number of virtual machines to create"
private: no
default: 1
- name: "vm_size"
prompt: "Choose a size for azure virtual machines"
private: no
default: "Standard_B2s"
- name: "managed_disk_type"
prompt: "Choose a type for azure virtual disks"
private: no
default: "Standard_LRS"
- name: "admin_username"
prompt: "Choose an admin username"
private: no
default: "zwindler"
- name: "admin_pub_path"
prompt: "Where can I find the admin public key ?"
private: no
default: "~/.ssh/id_rsa.pub"
- name: "local_ip"
prompt: "your local IP address (skip if you don't want to add 22 port to NSG)"
private: no
default: "skip"
- name: "virtualnetwork_cidr"
prompt: "Give a network CIDR for virtual network (large)"
private: no
default: "172.16.0.0/18"
- name: "subnet_cidr"
prompt: "Give a subnet of that network"
private: no
default: "172.16.1.0/24"
vars:
- resourcegroup_name: "{{project_prefix}}-rg"
- availabilityset_name: "{{project_prefix}}-avset"
- virtualnetwork_name: "{{project_prefix}}-vnet"
- subnet_name: "{{project_prefix}}-subnet"
- securitygroup_name: "{{project_prefix}}-nsg"
- vm_root_name: "{{project_prefix}}-vm"
- public_ip_name: "{{project_prefix}}-ip"
- nic_root_name: "{{project_prefix}}-nic"
tasks:
- name: "Create {{project_prefix}}-rg Resource Group"
azure_rm_resourcegroup:
name: "{{resourcegroup_name}}"
location: "{{location}}"
- name: "Create {{availabilityset_name}} Availability Set"
azure_rm_availabilityset:
name: "{{availabilityset_name}}"
location: "{{location}}"
resource_group: "{{resourcegroup_name}}"
sku: Aligned
- name: "Create {{virtualnetwork_name}} Virtual Network"
azure_rm_virtualnetwork:
name: "{{virtualnetwork_name}}"
resource_group: "{{resourcegroup_name}}"
address_prefixes_cidr:
- "{{virtualnetwork_cidr}}"
- name: "create {{subnet_name}} Subnet in {{virtualnetwork_name}} for VMs"
azure_rm_subnet:
name: "{{subnet_name}}"
virtual_network_name: "{{virtualnetwork_name}}"
resource_group: "{{resourcegroup_name}}"
address_prefix_cidr: "{{subnet_cidr}}"
register: subnet
- name: "Create {{securitygroup_name}} security rules (if local IP address was given)"
azure_rm_securitygroup:
name: "{{securitygroup_name}}"
resource_group: "{{resourcegroup_name}}"
purge_rules: yes
rules:
- name: 'AllowSSHFromYourOwnInternetIP'
protocol: 'Tcp'
source_address_prefix: "{{local_ip}}"
destination_port_range: 22
access: Allow
priority: 1000
direction: Inbound
when: local_ip | ipaddr
- name: "Create a {{nic_root_name}}X network interface for each VM"
azure_rm_networkinterface:
name: "{{nic_root_name}}{{item}}"
resource_group: "{{resourcegroup_name}}"
virtual_network: "{{virtualnetwork_name}}"
subnet_name: "{{subnet_name}}"
security_group: "{{securitygroup_name}}"
ip_configurations:
- name: "ipconfig"
public_ip_address_name: "{{public_ip_name}}"
primary: True
with_sequence: count="{{instances_number}}"
- name: "Create {{vm_root_name}}X VM with existing NIC"
azure_rm_virtualmachine:
resource_group: "{{resourcegroup_name}}"
name: "{{vm_root_name}}{{item}}"
vm_size: "{{vm_size}}"
managed_disk_type: "{{managed_disk_type}}"
admin_username: "{{admin_username}}"
availability_set: "{{availabilityset_name}}"
ssh_password_enabled: false
ssh_public_keys:
- path: "/home/{{admin_username}}/.ssh/authorized_keys"
key_data: "{{lookup('file', '{{admin_pub_path}}') }}"
network_interface_names: "{{nic_root_name}}{{item}}"
image:
offer: UbuntuServer
publisher: Canonical
sku: '16.04.0-LTS'
version: latest
with_sequence: count="{{instances_number}}"

For those of you that want to be spare the copy paste, this playbook can also be found on this Github repository

github.com/zwindler/ansible-deploy-azure

You only have to run it:

pip install --user netaddr
echo localhost > hosts
ansible-playbook -i hosts azure-deploy.yml
[...]
TASK [Create test-vmX VM with existing NIC] *****************************************
changed: [localhost] => (item=1)
PLAY RECAP **************************************************************************
localhost : ok=1 changed=7 unreachable=0 failed=0

Final words

Where are we now?

Now, we have a playbook, deploying VMs on Azure, but not only!

These machines are preinstalled with a Linux OS, preconfigured with your own SSH key and user. Only your local IP address can access it through SSH, all the VMs are in a single private virtual network; All the VMs are spread evenly accross Azure hypervisors to avoid SPOF by adding the VMs in an AVSET (availability set).

All of this could have been done by someone else by hand, with a little time. But not you. You, you used this playbook to do it all with only one command ;)

Aren’t you badass?

Deploy VMware virtual machines with Ansible

Tue, 26 Sep 2017 11:30:00 +0000

Why would I deploy virtual machines with Ansible

[Update]I wrote another article about [VM automation, for Azure this time. Check it out!](/2018/08/14/your-first-vm-in-azure-deployed-with-ansible-of-course/[/Update]

[French readers]La version française de l’article « Déployer des machines virtuelles VMware avec Ansible) est disponible à l’adresse suivante[/French readers]

Since a few month, I automate all that can be automated with Ansible (FR). Still, there was something that I still had to do manually and it bothered me (a lot).

Until then, I used Ansible both as a deployment tools (to install things on my machines) and also as a « Configuration Management » tool.

Let me explain :

From a virtual machine template, I can deploy default configuration (depending of the machine role). I mean by configuration, the first sense of the term like configuration files, but also package installations.
If I need to modify something on a subset of (or all!) my servers, I can push it on them with one command line.
Should something/someone modify something on the server managed configuration, I can correct this with my Ansible magic wand by just playing all my playbooks on all the servers once in a while. All « non compliant » servers will be automatically corrected WITHOUT TOUCHING those that ARE compliant.

For the last part, it’s not really « magic ». Playbooks have to be written while keeping I mind that they must be « idempotent ». This is really important and that’s why I had to develop my own Ansible module for Centreon.

Still, I was frustrated because one last step was still missing: Provisioning VMs directly from Ansible!

What is the problem?

2 obstacles prevented me from doing it.

Gather facts

First, Ansible works with an inventory. If you try to tell Ansible that you want to create new machines and you add them to the inventory, the first thing Ansible will try to do is to connect to them… before they exists!

ansible-playbook -l 192.168.100.103 zwindler_deploy.yml
PLAY [xxxxxxaaaaaaaaaaa] ********************************************************************************************************************************************************************
TASK [Gathering Facts] **********************************************************************************************************************************************************************
fatal: [192.168.100.103]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 192.168.100.103 port 22: No route to host\r\n", "unreachable": true}
PLAY RECAP **********************************************************************************************************************************************************************************
192.168.100.103 : ok=0 changed=0 unreachable=1 failed=0

We need to tell Ansible not to connect, at least at first.

You can do that by using the option gather_facts: false, which, you guessed, tells Ansible not to gather facts…

But that’s still not enough. If you try to run the playbook with just this option, Ansible will still try to connect to the yet to be created server, failing.

Most of the time, people use the delegate_to option, which tells Ansible to connect to another server for one specific task. I had used it previously to feed informations in Centreon with CLAPI.

Here, if you need to run several tasks and none of them have to be executed on the remote machine (that we are trying to deploy), you can just specify at the head of the playbook that all tasks are to be executed from the local machine, with connection: local :

gather_facts: false
connection: local

To sum this up, with these 2 options, you can tell Ansible not to « gather facts » as a preliminary step of every playbook, and to execute everything locally, thus avoiding the failing connection to the yet inexistant servers.

Problem solved.

vsphere_guest versus vmware_guest

My second issue was that 2 groups of modules were available in Ansible core, and one of them was abandoned. Of course, I was using the oldest abandoned one, because it’s the one that posts up in all the tutorials/examples I could find on Internet ;-) .

You can find it here vsphere_guest. It « works », and it has been integrated to the core way sooner (v 1.6) and needs the python dependancy pysphere (not maintained since 2013)!

There is no reason « not » to use vsphere_guest to provision your virtual machine. Latest version of this module works for vSphere v5 and should work for 6.X. A good example of what can be done with this module can be found on this nice blog-post from EverythingShouldBeVirtual.

Yet, you won’t be able to:

Modify the virtual machine (like vCPU, …) if you generate it from a vm template
Customize it on the OS level (change hostname, IP, …)

I massively use templates so this was clearly an issue. My typical usecase is this : I deploy a virtual machine from a given template (depending which type of service I want) and then finalize customisation through Ansible playbooks.

In this case, I would have to deploy my virtual machine, and still have to go into them to modify them (like hostname and IP) before playing my playbooks to customise them.

Hopefully, people at VMware didn’t stop at VIPerl and PowerCLI. It’s been a few years than there is a Python module, maintained by VMware, and regularly updated to take advantage of the new API functionnalities: pyvmomi.

From there, a new Ansible module, taking advantage of this python API, you have much more possibilities.

Let there be light

The list of Ansible modules taking advantage from pyvmomi is quite impressive. You can find them, they all start with vmware_ (you can find the list on the official Ansible documentation, VMware section).

Two of them particularly got my attention (but there are a lot more!):

vmware_guest can help manage virtual machines
vmware_vm_shell can execute commands directly inside the guest OS, provided it executes VMware Tools

Prerequisites

To execute this modules, you need:

pyvmomi and its dependancies (python 2.6+ and sphinx)

pip install pyvmomi sphinx

Ansible 2.2 minimum (but you’d better have 2.3 or even 2.4!)

~~On paper, the module « works » with 2.2, the actual version on stable depots. But most interesting functions appear in 2.3 so for now you have to compile sources!~~~~

The actual version on RHEL/CentOS 7 is 2.3.X. You don’t need to compile sources anymore, UNLESS you need the very latest additions which can be found in 2.4. I give you the necessary steps at the end of the article.

And now, the playbook !

The following playbook will:

ask you for a priviledged account for vCenter
deploy a virtual machine on the vCenter from a template

This deployment needs some variables that can be given directly from the Ansible host inventory like:

virtual disk size
vCPU number
vRAM quantity
target ESXi and target datasotre
VM IP address and hostname

The virtual machine will be deployed as expected, except for hostname and IP. Those 2 parameters will be modified after first boot by VMware Tool.

cat zwindler_vmware_add_normal.yml
---
- hosts: all
gather_facts: false
connection: local
vars_prompt:
- name: "vsphere_password"
prompt: "vSphere Password"
- name: "notes"
prompt: "VM notes"
private: no
default: "Deployed with ansible"
tasks:
# get date
- set_fact: creationdate="{{lookup('pipe','date "+%Y/%m/%d %H:%M"')}}"
# Create a VM from a template
- name: create the VM
vmware_guest:
hostname: '{{ vsphere_host }}'
username: '{{ vsphere_user }}'
password: '{{ vsphere_password }}'
validate_certs: no
esxi_hostname: esxi_server
datacenter: 'ZWINDLER'
folder: A_DEPLOYER
name: '{{ inventory_hostname }}'
state: poweredon
guest_id: rhel7_64Guest
annotation: "{{ notes }} - {{ creationdate }}"
disk:
- size_gb: 150
type: thin
datastore: '{{ vsphere_datastore }}'
networks:
- name: server_network
ip: '{{ custom_ip }}'
netmask: 255.255.252.0
gateway: 192.168.100.1
dns_servers:
- 192.168.100.10
- 192.168.101.10
hardware:
memory_mb: 4096
num_cpus: 2
customization:
dns_servers:
- 192.168.100.10
- 192.168.101.10
domain : zwindler.fr
hostname: '{{ inventory_hostname }}'
template: tmpl-rhel-7-3-app
wait_for_ip_address: yes
- name: add to ansible hosts file
lineinfile:
dest: /etc/ansible/hosts
insertafter: '^\[{{ ansible_host_group }}\]'
line: '{{ item }}'
with_items: '{{play_hosts}}'
run_once: true

From there… well, you can do anything that pops in your mind!

Useful links

« pause » module might be useful in some cases when you really have to wait some time. I don’t use it and I wouldn’t even consider it for production.

Pause Module

Also, I might write the same article with the Proxmox VE module. Stay tuned!

Proxmox Kvm Module

New functionnalities

Until 2.3~~ 2.4 becomes available in your OS repository, you might encounter this error if you use functions that aren’t included (yet) in your own version of Ansible (linked_clone or snapshot_src released in 2.4 for example) :

ansible-playbook zwindler_vmware_add_micro.yml
vSphere Password:
VM notes [Deployed with ansible]:
PLAY [ansible_node] **********************************************************
TASK [create the VM] ***********************************************************
fatal: [ansible_node]: FAILED! => {"changed": false, "failed": true, "msg": "unsupported parameter for module: linked_clone"}
PLAY RECAP *********************************************************************
ansible_node : ok=0 changed=0 unreachable=0 failed=1

Should you need a « yet to be released » functionnality, you can compile the latest version of Ansible from the sources (2.4 in my example here):

git clone https://github.com/ansible/ansible
cd ansible
make
make install
ansible --version
ansible 2.4.0

Create a plugin for Thruk using Thruk API, and work around permission issues

Tue, 19 Aug 2014 09:13:19 +0000

A friend of mine came to me with a problem. During one of his Shinken/Thruk deployment, he came with the idea of writing a plugin that could take advantage of Thruk statistics to graph the load of the monitoring engine.

Though you can get this information from Nagios itself through nagiostats, it’s a bit more difficult with the couple Thruk/Shinken. You could emulate Nagios by adding the Shinken old-cgi legacy module, or you could an authenticated wget the web page of Thruk and parse it, but let’s face it, these « solutions » aren’t pretty. I guess that the prettiest one should be to plug into Shinken, and maybe I’ll cover this later, because I think it’s the best solution, but this may need a little « dev effort ».

In order to avoid some useless effort, my friend asked Thruk developers how to do it the way Thruk does it. The main dev guided him to the « thruk » command line, which basically just generates the web page, but in a cleaner way as this is done internaly. You can get more information about Thruk command line with a simple « thruk –help » in your shell.

With this information, my friend developped a perl script (because he is a perl addict ;-) ) to parse this page and get the statistics he needed. But when he deployed it *shock*: he couldn’t see a thing INSIDE the Thruk console.

The main problem here isn’t the fact that the script doesn’t seem to work, but rather the lack of error output.

When first idea was to add the shinken user in apache group. This did help to make it work for the shinken user in shell, but for the web page of shinken/thruk it didn’t help.

To reproduce, I wrote a really simple script

#/bin/perl
#Print your page with thruk CLI, and display error output with 2>&1
print `/usr/bin/thruk -A thrukadmin extinfo.cgi?type=4 2>&1 `;

Though now the output was correct (lots of generated HTML jibber jabber) in the shell, the error is still displayed in Thruk

open file /etc/thruk/thruk_local.conf failed (id: uid=500(shinken) gid=500(shinken)
groups=500(shinken),0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023,
pwd: /var/run/shinken): at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 272.

AH-AH ! So, basically what I did was to add this at the first line of my script, to help me understand those groups/permissions issues.

#/bin/perl
print `id shinken`;
#Print your page with thruk CLI, and display error output with 2>&1
print `/usr/bin/thruk -A thrukadmin extinfo.cgi?type=4 2>&1 `;

Here is the output

uid=500(shinken) gid=500(shinken) groups=500(shinken),48(apache)
open file /etc/thruk/thruk_local.conf failed (id: uid=500(shinken) gid=500(shinken)
groups=500(shinken),0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023,
pwd: /var/run/shinken): at /usr/share/thruk/lib/Thruk/Utils/CLI.pm line 272.

This is what helped me understand what was really happening here. The groups were shinken and root, but there where no mention of apache, even though I had just added apache to shinken and restarted the whole lot.

In fact, the groups are set at another level : during the deamonization. By reading the files, I managed to find 3 places where shinken can set it’s own group :

/etc/default/shinken, but only used for run and var directories creation (first launch)
/etc/shinken/shinken.cfg, only the arbiter
/etc/daemons/*.ini, for the daemons

You need to have read access for these 3 files (for my setup anyway)

ll /etc/thruk/thruk*
-rw-r--r--. 1 apache apache 25933 17 mai 19:13 /etc/thruk/thruk.conf
-rw-rw----. 1 apache apache 671 18 août 15:36 /etc/thruk/thruk_local.conf
ll /var/lib/thruk/secret.key
-rw-r-----. 1 apache apache 32 13 août 13:13 /var/lib/thruk/secret.key

Opening theses file to « other » is obviously not an option, and I not a fan of changing the group of these conf files, even though this can/will work.

So, as of today, my favorite working solution (I’m still searching for a better one, and maybe I’ll switch to Shinken integration instead of Thruk) is to change shinken poller group in poller.ini to apache and restart the daemon. This way, the poller and the associated plugins can access those 3 files and your Thruk API will work in Thruk console as it does in the shell.