CloudFlare Access and Tunnel Woes

Some of the issues I encountered with these CloudFlare products

In my previous post, I talked about how I had this problem statement around making my services more universally accessible to me in the event of traveling somewhere new, or even if I’m using my VPN and still want that service I was using available. When I predominately used EC2, there was APIs that I could hit to do my firewall updates for me, but as I moved on to bigger things like my bare metal server, I lost that feature. I mentioned how I was looking at CloudFlare Access and CloudFlare Tunnel as a ‘Zero Trust’ based solution to my problem, but I touched on some issues I came across that ultimately made the products a no go while also making myself consider my relationship with CloudFlare as a services provider. Herein I go into some details on what I really mean when I said those things.

For starters, let me just say that I do love CloudFlare. Their core product of CDN, DDoS protection, reverse proxy, etc is simply amazing and does exactly what it says on the tin. The other products they launched since then either to hook into that service or standalone, are also noteworthy such as CloudFlare Pages or CloudFlare Workers. I feel that over time, Access and Tunnel will reach the same stature of Pages and Workers, but for right now, it just didn’t seem like the right time?

The entry level, hello world tutorials all made relative sense to me and I got things up and running in that context pretty quickly. I got to say the novelty of having a service that’s running, but on a completely firewalled machine and accessible was amazing. When I added the identity features of Access, I was just nerding out even more when thinking about all the possibilities that I could do. Once I got the basics down, I went down a route of slowly adding some services I was starting to run on Endeavour (which I’ll share in future posts) and it all ‘just works’. But, soon enter the stick that broke the camels back - GitLab.

I was toying with the idea of self hosting Gitlab, which is very easy to stand up but is a beast of a software tool. I’ll touch on my reasons in more detail in the future, but a very high level reason for today is that I wanted the all-in-one-ness it provided with Git, CI/CD, etc. CloudFlare have this rather long tutorial on ‘Zero Trust with Gitlab’. It basically touches on getting Gitlab installed on a machine, configuring Tunnel on that machine, getting Access configured with said tunnel and away you go. Following this, I got pretty far, so far as to having the UI accessible over the web. But obviously when you’re talking about Git, you’re gonna want to clone some code. HTTP cloning is great but as any dev knows, becomes annoying entering your password every time unless you configure a credential manager. So, the tutorial touched on configuring SSH with Tunnel, which they also had a standalone tutorial for. So far, so good I thought.

Now how this worked was fairly straight forward if you know SSH configuration commands. This was just adding a host entry that used ‘ProxyCommand’ to invoke cloudflared (the local binary for Tunnel) and to open your browser to authenticate via Access. Here’s the thing, that did not work for me! I would constantly get this issue around websockets being closed, or something to that effect. So I did what anyone would do and start to search for solutions. The issues I found that experienced this issue, were in a similar vein of machine access protocols, such as RDP. But either resolved it without posting an update (bold, very bold!) or resolved it with a setting they had turned off, but I already had turned on (eg, “turn on websockets” but I already have websockets turned on. Turning it on and off again didn’t work either >.>) so it became very painful. It was also not obvious how to increase log verbosity so I could potentially narrow my search, so I resolved to contact CloudFlare support.

Now at the time, I’m on the Pro plan with CloudFlare, so I do have access to their support team. I opened my ticket with as much detail as I could provide, my Tunnel config file, logs I could generate, etc. What followed was the usual, canned response asking for the information I already provided (pain), once they had that, the back and forth over several days began. I never really had a consistent support engineer, it felt like the buck was being passed around on every response. I did learn how to increase my log verbosity, but the additional info provided seemed pointless. All the while, things dragged on, slowing my progress in getting tools stood up. Ultimately I began to just take my applications off of Tunnel and Access.

There was also a moment where they decided to close my ticket on me, which was not ideal as you can imagine! The overall experience was just not nice and while I know that I’m not some large enterprise with a TAM, I should at least get a decent experience where I feel we are making headway in resolving the issue. Ultimately, I never felt that. Towards the end, I got told that some docs were updated that the websockets issue might be caused by their new, Super Bot Fight Mode service. But, turning that off did not resolve things either and I just replied saying “Thanks, but I’m done with this, all the best”. It was a good six to eight weeks of my life and I had enough.

So, I canceled my pro plan and resolved to start moving services away from CloudFlare again. Most likely, I will be heading towards AWS again with CloudFront. Ultimately, I need to vote with my wallet, as all of you should do when it comes to what providers you use. It’s going to be a tall order moving services, but I know I’ll get through it eventually. My next post will talk about the final nails in the coffin for CloudFlare, for me and what solutions I might have arrived at already for CloudFlare service replacements.