OpenStack Certificates with KubeOne

Currently I have set-up a small test environment where I have deployed terraform and kubeone v1.0.0-beta.2 from https://github.com/kubermatic/kubeone/releases/download/v1.0.0-beta.2/kubeone_1.0.0-beta.2_linux_amd64.zip.

My initial goal is to see how Kubeone works in order to deploy a Kubernetes management cluster. In the current state of my test environment, the Kubeone install command fails because the x509 certtificate from the OpenStack identity API is signed by an unkown authority. (I am using an Openstack cloud provider)

This is actually coming from kubeadm that does not have an ‘insecure’ or ‘verify’ - false option when trying to access the OpenStack identity API. Does Kubeone have any options to pass on some sort of parameter to kubeadm so that it does not try to verify the OpenStack certificate?

Hello,

Thank you for reporting the issue! Can you please provide us logs relevant to the errors you’re seeing?

It’s a little bit strange for kubeadm to try access the OpenStack Identity API. It can be done by kube-controller-manager, but that shouldn’t affect kubeadm’s ability to provision the cluster.

To get more detailed logs, you can run kubeone install with -v option.

Hello,

Here are the relevant log entries (ran kubeone command with -v 5 and -d):

[IP_ADDR] I0708 14:43:35.255502   16843 round_trippers.go:443] GET https://<IP_ADDR>:6443/healthz?timeout=10s  in 0 milliseconds                                                                                                    
[IP_ADDR] I0708 14:43:35.755494   16843 round_trippers.go:443] GET https://<IP_ADDR>:6443/healthz?timeout=10s  in 0 milliseconds                                                                                                    
[IP_ADDR] couldn't initialize a Kubernetes cluster                                                                                                                                                                                      
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase                                                                                                                                                    
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114                                             
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234                                                  
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422                                                  
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207                                                  
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147                                                                    
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826                                                       
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914                                                       
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864                                                       
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app.Run
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50                                                                      
[IP_ADDR] main.main
[IP_ADDR]         _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
[IP_ADDR] runtime.main
[IP_ADDR]         /usr/local/go/src/runtime/proc.go:203                                                                                                                                                                                
[IP_ADDR] runtime.goexit                                                                                                                                                                                                               
[IP_ADDR]         /usr/local/go/src/runtime/asm_amd64.s:1357
[IP_ADDR] error execution phase wait-control-plane
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235                                                  
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422                                                  
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207                                                  
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147                                                                    
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute                                                                                                                                                           
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826                                                       
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914                                                       
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864                                                       
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app.Run
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50                                                                      
[IP_ADDR] main.main                                                                                                                                                                                                                    
[IP_ADDR]         _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
[IP_ADDR] runtime.main
[IP_ADDR]         /usr/local/go/src/runtime/proc.go:203
[IP_ADDR] runtime.goexit
[IP_ADDR]         /usr/local/go/src/runtime/asm_amd64.s:1357
[IP_ADDR] error execution phase wait-control-plane
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
[IP_ADDR] k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
[IP_ADDR] k8s.io/kubernetes/cmd/kubeadm/app.Run
[IP_ADDR]         /workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
[IP_ADDR] main.main
[IP_ADDR]         _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
[IP_ADDR] runtime.main
[IP_ADDR]         /usr/local/go/src/runtime/proc.go:203
[IP_ADDR] runtime.goexit
[IP_ADDR]         /usr/local/go/src/runtime/asm_amd64.s:1357
[IP_ADDR] [kubelet-check] It seems like the kubelet isn't running or healthy.
[IP_ADDR] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[IP_ADDR]
[IP_ADDR]         Unfortunately, an error has occurred:
[IP_ADDR]                 timed out waiting for the condition
[IP_ADDR]
[IP_ADDR]         This error is likely caused by:
[IP_ADDR]                 - The kubelet is not running
[IP_ADDR]                 - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
[IP_ADDR]
[IP_ADDR]         If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
[IP_ADDR]                 - 'systemctl status kubelet'
[IP_ADDR]                 - 'journalctl -xeu kubelet'
[IP_ADDR]                                                                                                                                                                                                                      [59/1892]
[IP_ADDR]         Additionally, a control plane component may have crashed or exited when started by the container runtime.
[IP_ADDR]         To troubleshoot, list all containers using your preferred container runtimes CLI.
[IP_ADDR]
[IP_ADDR]         Here is one example how you may list all Kubernetes containers running in docker:
[IP_ADDR]                 - 'docker ps -a | grep kube | grep -v pause'
[IP_ADDR]                 Once you have found the failing container, you can inspect its logs with:
[IP_ADDR]                 - 'docker logs CONTAINERID'
[IP_ADDR]
WARN[14:43:51 UTC] Task failed, error was: Process exited with status 1
WARN[14:43:56 UTC] Retrying task…
INFO[14:43:56 UTC] Initializing Kubernetes on leader…
INFO[14:43:56 UTC] Running kubeadm…                              node=<IP_ADDR>
[IP_ADDR] + export PATH=/home/centos/.local/bin:/home/centos/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/opt/bin
[IP_ADDR] + PATH=/home/centos/.local/bin:/home/centos/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/opt/bin
[IP_ADDR] + [[ -f /etc/kubernetes/admin.conf ]]
[IP_ADDR] + sudo kubeadm --v=6 token create ybs1sq.xu0cwq53gs19u96z --ttl 1h0m0s
[IP_ADDR] I0708 14:44:03.452021   17587 token.go:121] [token] validating mixed arguments
[IP_ADDR] I0708 14:44:03.452177   17587 token.go:130] [token] getting Clientsets from kubeconfig file
[IP_ADDR] I0708 14:44:03.452205   17587 cmdutil.go:79] Using kubeconfig file: /etc/kubernetes/admin.conf
[IP_ADDR] I0708 14:44:03.453415   17587 loader.go:375] Config loaded from file:  /etc/kubernetes/admin.conf
[IP_ADDR] I0708 14:44:03.454881   17587 token.go:243] [token] loading configurations
[IP_ADDR] I0708 14:44:03.455240   17587 interface.go:400] Looking for default routes with IPv4 addresses
[IP_ADDR] I0708 14:44:03.455260   17587 interface.go:405] Default route transits interface "eth0"
[IP_ADDR] I0708 14:44:03.455389   17587 interface.go:208] Interface eth0 is up
[IP_ADDR] I0708 14:44:03.455444   17587 interface.go:256] Interface "eth0" has 2 addresses :[<IP_ADDR>/24 fe80::f816:3eff:fe3e:b5b5/64].
[IP_ADDR] I0708 14:44:03.455480   17587 interface.go:223] Checking addr  <IP_ADDR>/24.
[IP_ADDR] I0708 14:44:03.455487   17587 interface.go:230] IP found <IP_ADDR>
[IP_ADDR] I0708 14:44:03.455496   17587 interface.go:262] Found valid IPv4 address <IP_ADDR> for interface "eth0".
[IP_ADDR] I0708 14:44:03.455501   17587 interface.go:411] Found active IP <IP_ADDR>
[IP_ADDR] W0708 14:44:03.455671   17587 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[IP_ADDR] I0708 14:44:03.455684   17587 token.go:255] [token] creating token
[IP_ADDR] I0708 14:44:03.456020   17587 round_trippers.go:443] GET https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-ybs1sq?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:44:03.458052   17587 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 1 milliseconds
[IP_ADDR] I0708 14:44:08.458802   17587 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:44:18.459557   17587 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:44:38.460589   17587 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:45:18.461480   17587 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] timed out waiting for the condition
WARN[14:45:11 UTC] Task failed, error was: Process exited with status 1
WARN[14:45:21 UTC] Retrying task…
INFO[14:45:21 UTC] Initializing Kubernetes on leader…
INFO[14:45:21 UTC] Running kubeadm…                              node=<IP_ADDR>
[IP_ADDR] + export PATH=/home/centos/.local/bin:/home/centos/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/opt/bin
[IP_ADDR] + PATH=/home/centos/.local/bin:/home/centos/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/opt/bin
[IP_ADDR] + [[ -f /etc/kubernetes/admin.conf ]]
[IP_ADDR] + sudo kubeadm --v=6 token create ybs1sq.xu0cwq53gs19u96z --ttl 1h0m0s
[IP_ADDR] I0708 14:45:28.531211   17720 token.go:121] [token] validating mixed arguments
[IP_ADDR] I0708 14:45:28.531272   17720 token.go:130] [token] getting Clientsets from kubeconfig file
[IP_ADDR] I0708 14:45:28.531289   17720 cmdutil.go:79] Using kubeconfig file: /etc/kubernetes/admin.conf
[IP_ADDR] I0708 14:45:28.532156   17720 loader.go:375] Config loaded from file:  /etc/kubernetes/admin.conf
[IP_ADDR] I0708 14:45:28.533216   17720 token.go:243] [token] loading configurations
[IP_ADDR] I0708 14:45:28.533444   17720 interface.go:400] Looking for default routes with IPv4 addresses
[IP_ADDR] I0708 14:45:28.533449   17720 interface.go:405] Default route transits interface "eth0"
[IP_ADDR] I0708 14:45:28.533567   17720 interface.go:208] Interface eth0 is up
[IP_ADDR] I0708 14:45:28.533618   17720 interface.go:256] Interface "eth0" has 2 addresses :[<IP_ADDR>/24 fe80::f816:3eff:fe3e:b5b5/64].
[IP_ADDR] I0708 14:45:28.533644   17720 interface.go:223] Checking addr  <IP_ADDR>/24.
[IP_ADDR] I0708 14:45:28.533650   17720 interface.go:230] IP found <IP_ADDR>
[IP_ADDR] I0708 14:45:28.533656   17720 interface.go:262] Found valid IPv4 address <IP_ADDR> for interface "eth0".
[IP_ADDR] I0708 14:45:28.533661   17720 interface.go:411] Found active IP <IP_ADDR>
[IP_ADDR] W0708 14:45:28.533763   17720 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[IP_ADDR] I0708 14:45:28.533772   17720 token.go:255] [token] creating token
[IP_ADDR] I0708 14:45:28.534067   17720 round_trippers.go:443] GET https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-ybs1sq?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:45:28.534442   17720 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:45:33.535073   17720 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:45:43.537305   17720 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:46:03.538225   17720 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] I0708 14:46:43.539503   17720 round_trippers.go:443] POST https://<IP_ADDR>:6443/api/v1/namespaces/kube-system/secrets?timeout=10s  in 0 milliseconds
[IP_ADDR] timed out waiting for the condition
WARN[14:46:36 UTC] Task failed, error was: Process exited with status 1
Error: failed to install the cluster: failed to init kubernetes on leader: Process exited with status 1
Process exited with status 1
failed to init kubernetes on leader
github.com/kubermatic/kubeone/pkg/tasks.Tasks.Run
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/tasks/tasks.go:39
github.com/kubermatic/kubeone/pkg/cmd.runInstall
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/cmd/install.go:137
github.com/kubermatic/kubeone/pkg/cmd.installCmd.func1
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/cmd/install.go:94
github.com/spf13/cobra.(*Command).execute
        /home/marko/Projects/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:840
github.com/spf13/cobra.(*Command).ExecuteC
        /home/marko/Projects/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945
github.com/spf13/cobra.(*Command).Execute
        /home/marko/Projects/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
github.com/kubermatic/kubeone/pkg/cmd.Execute
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/cmd/root.go:52
main.main
        /home/marko/Projects/src/github.com/kubermatic/kubeone/main.go:24
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
failed to install the cluster
github.com/kubermatic/kubeone/pkg/cmd.runInstall
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/cmd/install.go:137
github.com/kubermatic/kubeone/pkg/cmd.installCmd.func1
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/cmd/install.go:94
github.com/spf13/cobra.(*Command).execute
        /home/marko/Projects/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:840
github.com/spf13/cobra.(*Command).ExecuteC
        /home/marko/Projects/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:945
github.com/spf13/cobra.(*Command).Execute
        /home/marko/Projects/pkg/mod/github.com/spf13/cobra@v0.0.6/command.go:885
github.com/kubermatic/kubeone/pkg/cmd.Execute
        /home/marko/Projects/src/github.com/kubermatic/kubeone/pkg/cmd/root.go:52
main.main
        /home/marko/Projects/src/github.com/kubermatic/kubeone/main.go:24
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373

As I understand, the task fails because the kubelet fails to start on the first master node.

And the kubelet fails with the following log:

-- The start-up result is done.
Jul 08 14:50:54 kubermatic-demo-cp-0.novalocal kubelet[18151]: I0708 14:50:54.071982   18151 server.go:417] Version: v1.18.0
Jul 08 14:50:54 kubermatic-demo-cp-0.novalocal kubelet[18151]: W0708 14:50:54.072359   18151 plugins.go:115] WARNING: openstack built-in cloud provider is now deprecated. Please use 'external' cloud provider for openstack: https://github/kubernetes/cloud-provider-openstack
Jul 08 14:50:54 kubermatic-demo-cp-0.novalocal kubelet[18151]: F0708 14:50:54.096712   18151 server.go:274] failed to run Kubelet: could not init cloud provider "openstack": Post https://<redacted>/v3/auth/tokens: x509: certificate signed by unknown authority
Jul 08 14:50:54 kubermatic-demo-cp-0.novalocal systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jul 08 14:50:54 kubermatic-demo-cp-0.novalocal systemd[1]: kubelet.service: Failed with result 'exit-code'.

This is actually coming from kubeadm (…)

Sorry - it’s actually the kubelet service that fails to start

@vbotez two points:

  • looks like you have very old binary release of kubeone (please grab the latest 1.0.0-beta.2
  • kubelet will use system trusted root certificates, so you need to place your custom’s CA cert into the system trusted root.

@kron4eg - I am using 1.0.0-beta.2. However, for OpenStack I have self signed certificates so a custom CA trust chain can’t really be verified.

It can, if you embed your custom CA into the system CA roots. Build new image that will have following steps executed:

  • copy your custom CA cert into /etc/pki/ca-trust/source/anchors
  • run update-ca-trust extract

(assuming your OS is CentOS)

use that new image for your nodes so kublets will trust your custom CA

Yes. Thank you. Building a new image that has the custom CA in the system CA roots should work.