Rachel's Yard

| A New Continuation
Categories Technical

Update 1: E3-12xx v4 has a lead time of 4 weeks from Asia via ACME, no source via Superbiiz

So I've been looking at Paperspace.io and Sixa.io for virtual desktop solutions, and it does look very appealing. However, is it possible to run it in a home-lab environment?

Currently, I have a XenServer running on my E5-1650v3 node (256GB) and Apache Guacamole for vdi. I do want to enable some sort of GPU capabilities (like for my Jetson TX2 training, but I will use Sixa.io for now; my workstation has a very shitty support for Linux). Looking at XenServer's HCL, and Intel seems to be a promising solution...

However, Intel is being a stupid asshole. E3-15xx v5 are available only in BGA. (WTF?) and E3-12xx v4 are not available anywhere except very special order. I was able to find v5 but only in Asia (none available in North America). What the fuck?

But, the Skull Canyon NUC does have a Iris Pro processor and unofficially supported by XenServer...

Here's the pricing as of the time of the writing:

pricing

On Tuesday half of the Internet broke because S3 was fucked. SlugSurvival happened to have a bug that needed a hotfix. However, since Docker Hub uses S3 as the backend, I could not push my images!

Outrageous!

Thus, I have to setup my own in my racks of servers.

Setup

  1. Minio
  2. Kubernetes
  3. mkuchin/docker-registry-web

As Fast As possible

First, setup the authentication service. Refers to mkuchin/docker-registry-web. You should have a public key and private key ready. kubectl create a secret for the public key part.

Then, spin up the registry:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
{
"apiVersion": "extensions/v1beta1",
"kind": "Deployment",
"metadata": {
"name": "kube-registry",
"namespace": "kube-system"
},
"spec": {
"replicas": 3,
"selector": {
"matchLabels": {
"app": "kube-registry"
}
},
"template": {
"metadata": {
"labels": {
"app": "kube-registry"
}
},
"spec": {
"volumes": [{
"name": "auth-signing",
"secret": {
"secretName": "auth-signing"
}
}],
"containers": [{
"name": "sysctl-buddy",
"image": "alpine:latest",
"command": [
"/bin/sh",
"-c",
"while true; do\n sysctl -w net.core.somaxconn=32768 > /dev/null 2>&1\n sleep 10\ndone\n"
],
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"privileged": true
}
}, {
"name": "registry",
"image": "registry:2.5.1",
"resources": {
"requests": {
"cpu": "200m",
"memory": "1024Mi"
},
"limits": {
"cpu": "2",
"memory": "2048Mi"
}
},
"env": [{
"name": "REGISTRY_LOG_LEVEL",
"value": "warn"
},
{
"name": "REGISTRY_HTTP_ADDR",
"value": ":5000"
},
{
"name": "REGISTRY_HTTP_HOST",
"value": ""
},
{
"name": "REGISTRY_HTTP_SECRET",
"value": ""
},
{
"name": "REGISTRY_STORAGE",
"value": "s3"
},
{
"name": "REGISTRY_STORAGE_DELETE_ENABLED",
"value": "true"
},
{
"name": "REGISTRY_STORAGE_S3_REGION",
"value": "us-east-1"
},
{
"name": "REGISTRY_STORAGE_S3_REGIONENDPOINT",
"value": "http://minio-docker"
},
{
"name": "REGISTRY_STORAGE_S3_BUCKET",
"value": "images"
},
{
"name": "REGISTRY_STORAGE_S3_ACCESSKEY",
"value": "docker"
},
{
"name": "REGISTRY_STORAGE_S3_SECRETKEY",
"value": "supersecret"
},
{
"name": "REGISTRY_STORAGE_S3_ENCRYPT",
"value": "false"
},
{
"name": "REGISTRY_STORAGE_S3_SECURE",
"value": "false"
},
{
"name": "REGISTRY_STORAGE_S3_CHUNKSIZE",
"value": "20971520"
},
{
"name": "REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR",
"value": "redis"
},
{
"name": "REGISTRY_REDIS_ADDR",
"value": "docker-redis-cache:6379"
},
{
"name": "REGISTRY_AUTH",
"value": "token"
},
{
"name": "REGISTRY_AUTH_TOKEN_REALM",
"value": "point to your auth server"
},
{
"name": "REGISTRY_AUTH_TOKEN_SERVICE",
"value": "must match"
},
{
"name": "REGISTRY_AUTH_TOKEN_ISSUER",
"value": "must match"
},
{
"name": "REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE",
"value": "/auth/auth.crt"
}
],
"ports": [{
"containerPort": 5000,
"name": "registry",
"protocol": "TCP"
}],
"volumeMounts": [{
"name": "auth-signing",
"mountPath": "/auth",
"readOnly": true
}]
}]
}
}
}
}

This also assumes that you have a Redis running somewhere in the same namespace.

Third, Profit (of course setup services and ingress as well).

Caveat

As of the time of this writing, registry:2 corrupts large layers (see my issue) when using 3rd-party S3. You need to use registry:2.5.1 or registry:2

Aug 2 2016

For the sake of humanity, let's point out the caveats first:

  1. If you ever change the CA or apiserver TLS certificate, remember to delete default secret tokens for ALL namespaces, and recreate services/pods. Your applications in each pods can access the API with a serviceaccount credentials, and it is only generated once. Therefore, if you change the TLS certs, the old secrets will be invalid.
  2. If you are running multiple master components, remember to add --apiserver-count=<count> flag in your kube-apiserver. Otherwise, apiserver will fight to get control of the service endpoints.

Now, let's get to the topic.

Billions of $$$ are awesome, but how do you invest? (joke

Docker is awesome (late to the party again), but how to manage?

TL;DR Kubernetes is a management tool (sort of) of containers.

I will just skip the introduction, since there are many articles out there already.

Installing Kubernetes

The best way of running Kubernetes is to deploy it on CoreOS, period.

https://coreos.com/kubernetes/docs/latest/getting-started.html

See, all other OSs are too heavyweight. CoreOS (other than SmartOS) is highly specialized to run containers, so there's that. Plus, folks at Quay.io is kind enough to have kubelet image ready. Kubelet is a binary that contains all the components that you will need to run Kubernetes (Golang is awesome), and Quay/CoreOS team makes it running in a rkt container, which makes updates/upgrades easy.

Of course, all components are stateless. Persistent states are stored in a etcd cluster (that seems to be the trend).

What's my environment?

An obligatory graph of architecture:

Kubernetes

Here are my cloud-config files for my deployment on OpenNebula: https://coreos-opennebula.s3.fmt01.sdapi.net/cloud-config/

What am I running?

Currently only a project written for an econ professor, but I will containerize more of my projects.

Ingress

The nginx controller on kubernetes/contrib kind of blows. So I (sort of) compiled the newest nginx and CHACHA20 ciphers:

Docker: https://hub.docker.com/r/zllovesuki/nginx-slim/

Git: https://git.fm/zllovesuki/nginx-slim/

You will need to recompile the controller with this tag.

next
1 2 3 ... 13
Weightless Theme
Rocking Basscss
RSS