Rachel's Yard

| A New Continuation

High availability is a topic that I really want to explore. My last project Dermail has failover in mind when designing the architecture. However, I'm more interested in the infrastructure side of things.

This is my attemp of trying CoreOS. I know, I'm very late to the party.

Environment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
System Information
Operating System Linux 3.16.0-4-amd64 x86_64
Model Supermicro Super Server
Motherboard Supermicro X10SDV-16C+-TLN4F
Processor Intel(R) Xeon(R) CPU D-1587 @ 1.70GHz @ 2.30 GHz
1 Processor, 16 Cores, 32 Threads
Processor ID GenuineIntel Family 6 Model 86 Stepping 4
L1 Instruction Cache 32.0 KB x 16
L1 Data Cache 32.0 KB x 16
L2 Cache 256 KB x 16
L3 Cache 24.0 MB
Memory 126 GB
BIOS American Megatrends Inc. 1.1a
Compiler Clang 3.3 (tags/RELEASE_33/final)

Storage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@puma:/tmp/igb-5.3.5.3/src# zpool status
pool: Storage
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
Storage ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD1003FZEX-00MK2A0_WD-WCC3F1VPH0LZ ONLINE 0 0 0
ata-WDC_WD1003FZEX-00MK2A0_WD-WCC3F1VPH3NT ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-WDC_WD1003FZEX-00MK2A0_WD-WCC3F7PDCJL6 ONLINE 0 0 0
ata-WDC_WD1003FZEX-00MK2A0_WD-WCC3FH3Y2ZT5 ONLINE 0 0 0
logs
nvme0n1 ONLINE 0 0 0
cache
ata-Samsung_SSD_850_PRO_128GB_S1SMNWAF720227X ONLINE 0 0 0

errors: No known data errors
root@puma:/tmp/igb-5.3.5.3/src# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Storage 1.81T 53.9G 1.76T - 0% 2% 1.00x ONLINE -

Wait, how are you running it on a single machine?

With 16 cores, 32 threads, and 128GB of RAM, this machine is a perfect home lab machine for testing. This machine will be running OpenNebula in a VM, as OpenNebula crashes on architecture newer than Haswell for some reasons.

OpenNebula and CoreOS

Compiling CoreOS Image for OpenNebula

This repo has a very good script to compile CoreOS for OpenNebula. You need packer for the script, and qemu.

Since the author is on vacation right now, I forked the repo and merge the PRs from other contributors, so that I can build the latest image. I also changed the channel from alpha to stable.

At the time of writing, CoreOS Stable channel is at version 1068.8.0. in builds/coreos-stable-1068.8.0-qemu/ we find coreos-stable-1068.8.0.

That script does not work. coreos-install with the install.yml did nothing.

Therefore, I re-created a similar script on my private git (or on github). You need Packer and QEMU. make latest to build the latest CoreOS image for OpenNebula.

This will make sure that OpenNebula contextualization works.

Your VM template will look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
CONTEXT = [
EC2_USER_DATA = "$EC2_USER_DATA",
NETWORK = "YES",
SET_HOSTNAME = "$NAME",
SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]" ]
CPU = "2"
DISK = [
IMAGE = "CoreOS Stable 1068.8.0" ]
FEATURES = [
ACPI = "yes",
APIC = "yes",
HYPERV = "yes",
PAE = "yes" ]
GRAPHICS = [
LISTEN = "0.0.0.0",
TYPE = "VNC" ]
HYPERVISOR = "kvm"
MEMORY = "8192"
NIC = [
NETWORK = "Internal" ]
NIC = [
NETWORK = "External" ]
NIC_DEFAULT = [
MODEL = "virtio" ]
OS = [
ARCH = "x86_64" ]
USER_INPUTS = [
EC2_USER_DATA = "M|text64|cloud-config" ]
VCPU = "4"

The image supports etcd2 configuration. However, you will use $COREOS_PUBLIC_IPV4 and $COREOS_PRIVATE_IPV4. For example, this is your cloud-config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#cloud-config

coreos:
units:
- name: etcd2.service
runtime: true
drop-ins:
- name: 10-oem.conf
content: |
[Service]
Environment=ETCD_ELECTION_TIMEOUT=1200
- name: reload.service
command: start
content: |
[Unit]
Description=reload systemd

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl daemon-reload
- name: start.service
command: start
content: |
[Unit]
Description=start etcd2

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl start etcd2

etcd2:
# generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
discovery: "https://discovery.etcd.io/<token>"
advertise-client-urls: "http://$COREOS_PUBLIC_IPV4:2379"
initial-advertise-peer-urls: "http://$COREOS_PRIVATE_IPV4:2380"
listen-client-urls: "http://0.0.0.0:2379,http://0.0.0.0:4001"
listen-peer-urls: "http://$COREOS_PRIVATE_IPV4:2380,http://$COREOS_PRIVATE_IPV4:7001"

fleet:
public-ip: "$COREOS_PRIVATE_IPV4"
metadata: "region=fmt01"

update:
reboot-strategy: "best-effort"

Spin Them Up

Sunstone

etcd2

(To be continued)

Weightless Theme
Rocking Basscss
RSS