source controller needs proxy to access github.com when running inside corporate network

[nab-gha]

In order to get the source controller to access a github.com repository from within my client's corporate network I needed to add the following environmental variables to the source-controller deployment.

    - name: HTTPS_PROXY
      value: http://http.proxy.abc.com:8000
    - name: NO_PROXY
      value: 10.0.0.0/8

Maybe gotk install could be enhanced to generate this?

[stefanprodan]

You could create a kustomize patch after running gotk install --export and add those values to all controllers.

[seaneagan]

Another way to make this easier: fluxcd/flux2#234

In which case --http-proxy could be an option to gotk create source git/helm

[davidkarlsen]

Can fluxcd/flux#2179 be re-implemented for fluxv2?
Just setting http_proxy won't cut it when you want to checkout over ssh (to avoid PATs and use deployment keys which are tied to ssh).

For flux v1 I installed it with values like:

ssh:
  known_hosts: "[ssh.github.com]:443 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=="
  config: |
    Host github.com
    ProxyCommand socat STDIO PROXY:proxy.fqdn.com:%h:%p,proxyport=8080
    User git
    Hostname ssh.github.com
    Port 443
    IdentityFile /etc/fluxd/ssh/identity

and need something similar in flux v2

[dvianello]

We're also blocked from migrating to v2 because of this.

And yes, http_proxy won't cut it, as we need to pass the same ProxyCommand @davidkarlsen is referring to.

[hiddeco]

Ability to configure HTTP/S and/or SSH client details is related to #93

[stealthybox]

ProxyCommand socat STDIO PROXY:proxy.fqdn.com:%h:%p,proxyport=8080

It might be possible to support this by modifying the source-controller /etc/hosts to point to a socat side-car or Service.

[davidkarlsen]

ProxyCommand socat STDIO PROXY:proxy.fqdn.com:%h:%p,proxyport=8080

It might be possible to support this by modifying the source-controller /etc/hosts to point to a socat side-car or Service.

I like this idea, nice and separated in a way.
So I tried this:

• run a container [1]
• add a svc for it [2]
• add hostentry for github.com [3] in src ctrl

but then src ctrl says:

{"level":"error","ts":"2021-01-28T21:37:58.369Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"test","namespace":"flux-system","error":"unable to clone 'ssh://git@github.com/acme/test-fluxv2', error: ssh: handshake failed: EOF"}

k get gitrepository test
NAME   URL                                                    READY   STATUS                                                                                                      AGE
test   ssh://git@github.com:443/acme/test-fluxv2.git   False   unable to clone 'ssh://git@github.com:443/acme/test-fluxv2.git', error: ssh: handshake failed: EOF   44m

[1]

args:
  - tcp-listen:8080,fork,reuseaddr
  - proxy:proxy.acme.org:github.com:443,proxyport=3128
command:
- socat

[2]

k get svc gh-proxy
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
gh-proxy   ClusterIP   10.201.37.75   <none>        22/TCP,443/TCP   60m

[3]

- hostnames:
  - github.com
  ip: 10.201.37.75

I think maybe the next hurdle is https://docs.github.com/en/github/authenticating-to-github/using-ssh-over-the-https-port

[davidkarlsen]

Update, I had missed an essential thing, the host is named ssh.github.com, when you run on port 443, we have a winner 🎉

k describe gitrepository test|tail -1
  Normal  info    75s                  source-controller  Fetched revision: main/6cb9fb47c005e5941233e7f69cb720c28455a89c

sidecar cmd:

k exec  svc/gh-proxy -- ps xa|grep soca
      1 ?        Ss     0:00 socat tcp-listen:8080,fork,reuseaddr proxy:proxy.acme.com:ssh.github.com:443,proxyport=8080

git repo url

url: ssh://git@github.com/acme/test-fluxv2.git

and the host entry for github.com

[stealthybox]

Thank you for proving this works at the pod infra level.

I like this solution.

Seems like it would be odd/complex to add this to GitRepository.spec as well as all the other Sources...

It's very coupled to the network details of the Cluster's environment -- very much an admin concern

Multiple tenants could transparently be using this proxy, and it would still make sense. (and be safe?)

User-facing Source configs between airgapped and cloud clusters can be the same or similar without being polluted with proxy details.

[stealthybox]

Using an actual side-car removes a hop to the Service Pod and reduces config, because you don't need the svc IP, it's just localhost.
This could stop unintentional cross-zone or inter-rack traffic.

The downside is that we don't run all of the flux controllers in the same process or Pod, so any controllers that need to talk directly to GitHub need their own socat side-car, whereas the Service Pod is reusable, even beyond Flux (ex: your Tekton jobs write artifacts or commit back to the repo -- they can use the proxy too)

[davidkarlsen]

Seems like it would be odd/complex to add this to GitRepository.spec as well as all the other Sources...

Yes, I don't think it belongs in the spec, since it is controller-scope'd (we redefine the host) - so two different gitrepo cr's will get the same treatment, given the same target host.

It's very coupled to the network details of the Cluster's environment -- very much an admin concern

Indeed, ideally it belongs in something like Istio (or even more ideally it's dealt with at infra networking level outside of the cluster handling it transparently - sadly we're not there/I can't influence that decision as of now).
Actually there are recent merges into Envoy to support this, but it's going to take time to land in Istio, if it ever does.

Multiple tenants could transparently be using this proxy, and it would still make sense. (and be safe?)

It should be safe, it's an opt-in new endpoint (opt-in by defining /etc/hosts entries), and all we are doing is providing an alternative path to an existing public endpoint on the internet. Additionally one can apply NetworkPolicies if wanted.

User-facing Source configs between airgapped and cloud clusters can be the same or similar without being polluted with proxy details.

Exactly, the last step is just howto be able to define hostAliases for the source-controller w/o hacking on the deployment post-intstall time, potentially loosing the config on upgrades etc.
One would then follow the same pattern for other sources, like HelmRepos etc, however, we can proxy these as an alternative.

[davidkarlsen]

Using an actual side-car removes a hop to the Service Pod and reduces config, because you don't need the svc IP, it's just localhost.
This could stop unintentional cross-zone or inter-rack traffic.

That's fixable by applying NetworkPolicies ?

The downside is that we don't run all of the flux controllers in the same process or Pod, so any controllers that need to talk directly to GitHub need their own socat side-car, whereas the Service Pod is reusable, even beyond Flux (ex: your Tekton jobs write artifacts or commit back to the repo -- they can use the proxy too)

How/for what reasons do the other controllers talk to github? (API?)

[stefanprodan]

How/for what reasons do the other controllers talk to github?

The image automation controller writes back to Git, see https://toolkit.fluxcd.io/guides/image-update/#configure-image-updates

[davidkarlsen]

so... how do we proceed from here?