haproxy-auth-request

HTTP access control using subrequests

Direct link to the repository for the impatient.

For a project of mine I needed to authenticate a medium number of vHosts behind an haproxy to the same group of users. I usually use TLS client certificates to protect internal services only I (or a small number of technically savy people) need to access. But they come with some serious issues in usablity, so I quickly disregarded them. Another popular choice is HTTP authentication (“.htaccess”), but that would require setting up and securely distributing credentials to the users that need to access the vHosts. On top of that the haproxy documentation discourages using hashed passwords, because the hash algorithms are designed to be CPU intensive and thus would block haproxy's event loop. I don't want to store unhashed passwords, even if they are coming from a secure random number generator. Setting up IP based protection or a VPN was out of question as well, because it either is insecure or too hard to set up for the target group of users.

Some time ago I learned about bitly/oauth2_proxy which does exactly what it's name implies: It verifies users against some OAuth 2 service, sets a cookie and tunnels all the requests to a configured upstream if the cookie is valid. As all the users already are part of an OAuth 2 provider this seemed to fit my needs, except for one thing: I did not want to send everything from the Internet, via haproxy, via oauth2_proxy to the backend services for these reasons:

  1. Configuring different upstreams is based on the path according to the README, I needed to route based on the Host.
  2. It does not allow for load balancing between multiple backends. I would have to put a load balancer behind oauth2_proxy, leading to even more possible points of failure.

Luckily oauth2_proxy also supports an endpoint that just returns whether a request should be allowed or not: I would be able to ask oauth2_proxy whether the request is good and perform the remaining delegation in haproxy. There is only one issue: haproxy supports nothing like nginx' auth_request. But it supports Lua - it should be possible to build something.

Meet Lua

Disclaimer: This post was written after the fact and thus the actual timeline of things does not match up. Partly to allow for an easier explanation, partly because I don't remember the actual progression of my Lua script.

All the code snippets in this post are licensed under the terms of the MIT license.

It was not my first time programming Lua, I built two small Lua programs into nginx before. It was my first time programming Lua in haproxy though. The first thing was to learn how exactly Lua was being called from haproxy:

The first Lua program

I searched for the documentation of Lua in haproxy and started reading. There are four main things I could register into haproxy:

action
Allows to modify requests.
converters
Allows to modify samples.
fetches
Allows to add samples.
service
Allows to answer requests.

At first I misunderstood what a service does and tried to use that one to handle the auth requests. But as I needed to modify requests and then pass them to a backend I needed an action:

auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn)
	-- code here
end)
haproxy.cfg
global
	lua-load auth-request.lua

frontend http
	mode http
	option httplog

	bind :::80 v4v6

	http-request lua.auth-request

haproxy would now call my function for each HTTP request. While looking around the Lua documentation I found the print_r function, which would make debugging easier, as I did not have to put the information into response headers. I promptly added it to my script. When running haproxy using the -d option it would print the parameter to my console.

The first HTTP request

Next thing was searching for a Lua HTTP library, I really did not want to implement HTTP on my own. I quickly found socket.http and installed it using apt-get install lua-socket. A quick test proofed successful: I was able to make HTTP requests from within my function:

auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn)
	local b, c, h = http.request {
		url = "http://127.0.0.1"
	}
	print_r(b)
	print_r(c)
end)

haproxy's Sockets

I however noticed the Socket class in haproxy's documentation before and thought that there must be a reason why it exists. Possibly the default Socket class would block haproxy's event loop. Luckily the developers of socket.http imagined that use case and support passing a custom constructor as create to the http.request function. I quickly added it and tried it out: Nothing happened. An strace confirmed that haproxy was not even connecting to the host. Poking around in the http.socket module revealed that commenting out the call to settimeout would allow it to set up the socket. I figured I would find out what haproxy's Socket is doing differently later:

auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn)
	local b, c, h = http.request {
		url = "http://127.0.0.1",
		create = core.tcp
	}
	print_r(b)
	print_r(c)
end)

Now haproxy was sending the HTTP request and even reading the response. First success! Unfortunately instead of the response code it now was printing: connection closed. to my console. By aging that message in haproxy's source code I could find that the issue must be somewhere inside the receive method. print_ring the results of the receive calls in socket.http lead me to the receiveheaders function. For some reason that line was returning an empty string instead of a valid HTTP header, it did however read the HTTP status line in receivestatusline successfully before. I figured to explicitly add the default parameter of "*l" to instruct it to read a line. With success.

haproxy must be incorrectly handling the default parameter, despite explicitely documenting that no parameter equals *l. As I did not want to patch all of http.lua I tried to find out whether I could monkey patch the Socket class inside of Lua:

auth-request.lua
function create_sock()
	local sock = core.tcp()

	sock.old_receive = sock.receive
	sock.receive = function(socket, pattern, prefix)
		local a, b
		if pattern == nil then pattern = "*l" end
		if prefix == nil then
			a, b = sock:old_receive(pattern)
		else
			a, b = sock:old_receive(pattern, prefix)
		end
		return a, b
	end

	return sock
end
core.register_action("auth-request", { "http-req" }, function(txn)
	local b, c, h = http.request {
		url = "http://127.0.0.1",
		create = create_sock
	}
	print_r(b)
	print_r(c)
end)

It now correctly printed the response code. I was able to make HTTP requests with haproxy's Socket class. On to find out why the call to settimeout inside socket.http caused the request to fail, despite settimeout working from within my Lua code. Poking around in the source code of socket and haproxy revealed that haproxy was returning 0, while the original socket class was returning 1. For my workstation I fixed the return code in haproxy and recompiled it. With success. For production I did not want to run a self compiled haproxy and thus monkey patched settimeout as well:

auth-request.lua
function create_sock()
	local sock = core.tcp()

	sock.old_receive = sock.receive
	sock.receive = function(socket, pattern, prefix)
		local a, b
		if pattern == nil then pattern = "*l" end
		if prefix == nil then
			a, b = sock:old_receive(pattern)
		else
			a, b = sock:old_receive(pattern, prefix)
		end
		return a, b
	end

	sock.old_settimeout = sock.settimeout
	sock.settimeout = function(socket, timeout)
		socket:old_settimeout(timeout)

		return 1
	end

	return sock
end
core.register_action("auth-request", { "http-req" }, function(txn)
	local b, c, h = http.request {
		url = "http://127.0.0.1",
		create = create_sock
	}
	print_r(b)
	print_r(c)
end)

The actual patch in haproxy probably occured later, as I fixed the bug in the receive function before fixing settimeout. I fixed IPv6 support first as well.

Making DNS requests

Lua in haproxy is documented to not support DNS requests. I did not want to hardcode an IP address in my configuration, however. While searching through haproxy's Lua documentation I noticed that I could get information about the backends using core.backends, with the Server class carrying a get_addr method. Would that give me the resolved IP address? It turned out that it does:

haproxy.cfg
global
	lua-load auth-request.lua

frontend fe
	mode http
	bind :::8080 v4v6

	http-request lua.auth-request

	default_backend be

backend be
	mode http
	server s example.com:80
auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn)
	local b, c, h = http.request {
		url = "http://" .. core.backends["be"].servers["s"]:get_addr(),
		create = create_sock
	}
	print_r(b)
	print_r(c)
end)

Better! While the name of the backend still is hardcoded the IP address is not. At first I used the txn:get_var("txn.foo") method to read a variable I set using http-request set-var("txn.foo") string("be"), but while making the bug fixes to haproxy I noticed that I was able to pass additional parameters to my Lua action:

haproxy.cfg
global
	lua-load auth-request.lua

frontend fe
	mode http
	bind :::8080 v4v6

	http-request lua.auth-request be

	default_backend be

backend be
	mode http
	server s example.com:80
auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn, be)
	local b, c, h = http.request {
		url = "http://" .. core.backends[be].servers["s"]:get_addr(),
		create = create_sock
	}
	print_r(b)
	print_r(c)
end, 1)

This later evolved into looping through all the servers in that backend and checking their health status.

I am now able to make HTTP requests to some easily changed backend. I now needed to return the response information back to haproxy:

Telling haproxy the result of the auth-request

Similarly to my first attempt of passing the backend name I could pass variables back to haproxy using txn:set_var("txn.foo", true). This was fairly straight forward:

auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn, be)
	txn:set_var("txn.auth_response_successful", false)
	local b, c, h = http.request {
		url = "http://" .. core.backends[be].servers["s"]:get_addr(),
		create = create_sock
	}
	if 200 <= c and c < 300 then
		txn:set_var("txn.auth_response_successful", true)
	end
end, 1)

Passing request headers

The only thing missing is passing all the request headers to my auth service (as it needed to know whether a specific cookie is set or not). This required me to convert from haproxy's header representation to the one of socket.http: haproxy passes header values as a table to support duplicate headers. socket.http only supports a single header name to string mapping. Luckily RFC 7230 allows me to support this without losing information:

A recipient MAY combine multiple header fields with the same field name into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field value to the combined field value in order, separated by a comma. The order in which header fields with the same field name are received is therefore significant to the interpretation of the combined field value; a proxy MUST NOT change the order of these field values when forwarding a message.
auth-request.lua
core.register_action("auth-request", { "http-req" }, function(txn, be)
	txn:set_var("txn.auth_response_successful", false)

	local headers = {}
	for header, values in pairs(txn.http:req_get_headers()) do
		for i, v in pairs(values) do
			if headers[header] == nil then
				headers[header] = v
			else
				headers[header] = headers[header] .. ", " .. v
			end
		end
	end

	local b, c, h = http.request {
		url = "http://" .. core.backends[be].servers["s"]:get_addr(),
		create = create_sock,
		headers = headers
	}
	if 200 <= c and c < 300 then
		txn:set_var("txn.auth_response_successful", true)
	end
end, 1)

Wrapping it up

Afterwards I added error handling to the script, a second parameter specifying the path to request on the authentication backend, checking of the status of the servers in the backend to support basic health checking and returning of a few more information in additional variables. The full script can be found on GitHub. It is then used like this:

haproxy.cfg
global
	lua-load /usr/share/haproxy/auth-request.lua

frontend http
	mode http
	option httplog

	bind :::80 v4v6

	acl  acme_challenge  path_beg       /.well-known/acme-challenge/
	acl  oauth_proxy     path_beg       /oauth2/

	http-request lua.auth-request oauth_proxy /oauth2/auth

	use_backend  certbot        if  acme_challenge
	use_backend  oauth_proxy    if  ! { var(txn.auth_response_successful) -m bool }
	use_backend  oauth_proxy    if  oauth_proxy
	use_backend  …              if  …

Debian Package

I am not a fan of manually placing files in the file system, that's why I wanted to pack the thing up into a Debian package. This is not my first Debian package, I built one to deploy configuration and files in my personal server. But for this I tried to “do it right”. My personal package is missing a description for example and has a few unnecessary files in it. Also it's a bunch of unrelated services munched together.

I followed the Debian New Maintainers' Guide to create a so-called “native” package. The difference, as I understand it, is that a native package does not ship with patchfiles.

As I only needed to install a single static file I did not need to make large changes to the autogenerated package. I added a Makefile with an install target and a README.md for documentation, but I would have created the latter anyways. The remaining changes boil down to removing a few files I did not need, filling out a few placeholders and adding the required dependencies.

You can view the changes I did after the automated generation of the package on GitHub. One thing I am not sure about is the Suggests dependency to haproxy. Should that be a hard Depends? While not strictly required (the file cannot do harm if haproxy is not installed) it just does not make sense without haproxy. I'm open for feedback from people more experienced with Debian packaging :-).

Final Words

This journey resulted in a total of 4 patches to haproxy to fix the bugs in the Lua interpreter I found:

  1. [PATCH 1/2] BUG/MINOR: lua: Fix default value for pattern in Socket.receive
  2. [PATCH 2/2] DOC: lua: Fix typos in comments of hlua_socket_receive
  3. [PATCH] BUG/MEDIUM: lua: Fix IPv6 with separate port support for Socket.connect
  4. [PATCH] BUG/MINOR: lua: Fix return value of Socket.settimeout

All of the patches have already been applied to the development version of haproxy. All of them have been backported to the upcoming haproxy 1.8.4 as well; all but the IPv6 to the 1.7 branch.

Overall the Lua API of haproxy is fairly easy to use. I was able to write the whole thing, including the patches to haproxy itself, in three afternoons, despite never having used Lua in haproxy before. The documentation is lacking in a few places (e.g. the undocumented additional parameters to an action), I hope to fix this in the future with more patches.

The script is now running in production, without any glaring issues so far, but I'd like to hear your feedback!