Node scaling not working after password change

I tried to scale up my k8s base cluster using the machine controller.
ks scale machinedeployments democluster-pool1 --replicas=9
however, the machine-controller fails with:
E0330 xxxxmachineset_controller.go:141] Failed to reconcile MachineSet "kube-system/democluster-pool1-67648bd9b4": failed to sync Machineset replicas: admission webhook "machine-controller.kubermatic.io-machines" denied the request: failed to default machineSpec: An error of type = InvalidConfiguration, with message = A request has been rejected due to invalid credentials which were taken from the MachineSpec: Authentication failed occurred; admission webhook "machine-controller.kubermatic.io-machines" denied the request: failed to default machineSpec:

So where is this hidden then?

  • the mc pod references the cloud-provider-credentials which I checked.
  • I also updated the cloud-config on the existing nodes with the new pw to be sure

but still authentication error. what can i do?
where are those machinespecs hidden?

So they are in the kubeone.yaml manifest and in the cloud-provider-credentials. Maybe elsewhere.
rerunning the kubeone apply seems to be the only way to distribute the correct one.

After I had issues with endpointslices - which would not cooperate - delete with mc service.

last are timeouts for starting the machines which can be controlled like this:

Final step is to integrate those fixes into the kubeone apply run… lets see…