h-app-roxy

Application server inside haproxy –

haproxy is “The Reliable, High Performance TCP/HTTP Load Balancer” according to the slogan on haproxy.org. I absolutely enjoy putting it in front of all the services I deploy. haproxy is absolutely rock-solid. The applications behind it? Not absolutely! Is there a better way to improve the reliability of your services than by putting them directly into haproxy and removing the unreliable backend from the equation?

In my previous post I explored how one can use Lua to intercept requests before they are being passed to a backend. This post will explore so-called “applets” or “services” to build an URL shortener backend into haproxy itself! As in the previous post the timeline is fictitious to allow for an easier read.

All the code snippets in this post are licensed under the terms of the MIT license.

The first applet

An URL shortener obviously needs a form for users to input their URLs. I copied the “Hello World” example from the official documentation and modified it to output a simple HTML page:

shorturl.lua
core.register_service("shorturl", "http", function(applet)
	applet:set_status(200)
	applet:add_header("content-type", "text/html")
	applet:start_response()
	applet:send([[
<html>
	<head><title>Short URL service</title></head>
	<body>
		<form action="." method="post">
			<input type="url" name="url" placeholder="URL" />
			<button type="submit">Submit</button>
		</form>
	</body>
</html>]])
end)
haproxy.cfg
global
	nbthread 1
	lua-load /usr/share/haproxy/shorturl.lua

frontend fe
	bind *:8080

	http-request use-service lua.shorturl

Processing POST requests

haproxy now was sending out my simple HTML page whenever I accessed the frontend. Next I needed to handle the POST request when a user submitted the form. Instead of using multiple applets and performing routing using ACLs in haproxy.cfg I opted to just pass everything to the Lua script and deciding what to do using Lua. That way I get a self contained application.

Adding a routing function

At first I used a simple if-elseif-else chain to perform the routing based on applet.path and applet.method, but this later evolved into a declarative routing table similar to Ruby on Rails, Laravel, Phoenix, express or whatever framework you prefer. I’m only showing the latter, because all of you know what an “if” is:

shorturl.lua
local function app(controllers)
	return function(applet)
		pattern_match = false
		for i, v in ipairs(controllers) do
			if applet.path:match("^" .. v["pattern"] .. "$") then
				pattern_match = true
				if applet.method == v["method"] then
					return v["controller"](applet, applet.path:match("^" .. v["pattern"] .. "$"))
				end
			end
		end

		if pattern_match then
			error(applet, 405)
		else
			error(applet, 404)
		end
	end
end

app is a function that takes a table of controllers and returns a function that can be passed to register_service. To perform the routing I loop over the controller table, check whether the pattern matches the requested path, then check whether the method matches the request method. If both match I call the controller function and pass the applet, as well as the groups in the matched pattern (the latter is required for the controller that performs the redirect to the target URL).

The error function is a helper function to return a response with a specific HTTP status code:

shorturl.lua
local function error(applet, number)
	applet:set_status(number)
	applet:start_response()
	applet:send(number)
end

The app is then registered like this:

shorturl.lua
core.register_service("redis", "http", app{
	{ pattern="/",      method="GET",  controller=getRoot     },
	{ pattern="/",      method="POST", controller=postRoot    },
	{ pattern="/(%w+)", method="GET",  controller=getShortUrl }
})

Actually processing the POST request

haproxy does not parse the HTTP request body itself. Thus I needed to parse the application/x-www-form-urlencoded request myself. While researching how to do this easiest in Lua I stumbled upon the Captures section in the Lua manual, which gives a working decode function. Perfect!

Now I just need to read and parse the body to get the required fields. Add in some validation of the Content-Type for good measure:

shorturl.lua
function postRoot(applet)
	if applet.headers["content-type"] == nil or #applet.headers["content-type"] > 1 or applet.headers["content-type"][0] ~= "application/x-www-form-urlencoded" then
		return error(applet, 415)
	end

	local body = applet:receive()
	local variables = decode(body)

	if variables["url"] == nil then
		return error(applet, 400)
	end
end

variables["url"] now contains the to-be-shortened URL.

Choosing a database

This one is easy: The haproxy documentation gives an example on how to use the lua-redis library with haproxy. Thus Redis it is. I augmented it with Thierry Fournier’s connection pool implementation. Note a small typo in there: r.release(conn) in renew should read r:release(conn).

To make the usage less verbose I added a helper function that aborts the haproxy request, if a call fails (while still making sure to give the connections back into the pool):

shorturl.lua
local redis = require("redis-pool").new("172.17.0.2", 6379, 20)

function call_redis(method, ...)
	local conn = redis:get(1000)

	if conn == nil then
		error("Could not get connection")
	end

	local client = conn.client

	if not pcall(client.ping, client) then
		redis:renew(conn)
		return call_redis(method, ...)
	end

	local result = table.pack(pcall(client[method], client, ...))

	if not table.remove(result, 1) then
		redis:renew(conn)
		error("Dead connection")
	end

	redis:release(conn)

	return table.unpack(result)
end

Storing the URL

This one is straight forward: Generate a random key and store the URL in Redis. Afterwards show the generated key to the user.

shorturl.lua
function postRoot(applet)
	if applet.headers["content-type"] == nil or #applet.headers["content-type"] > 1 or applet.headers["content-type"][0] ~= "application/x-www-form-urlencoded" then
		return error(applet, 415)
	end

	local body = applet:receive()
	local variables = decode(body)

	if variables["url"] == nil then
		return error(applet, 400)
	end

	while true do
		local key = math.random(1, 255)

		if call_redis("setnx", key, variables["url"]) then
			applet:set_status(200)
			applet:add_header("content-type", "text/html")
			applet:start_response()
			local response = [[
<html>
<head><title>Short URL service</title></head>
<body>
	<a href="%s">%s</a>
</body>
</html>]]
			applet:send(response:format(key, key))
			return
		end
	end
end

Performing the redirect

The last part is the redirect: Read the key from the parameter, look it up and perform redirect.

shorturl.lua
local redirect = function(applet, key)
	local value = call_redis("get", key)

	if value then
		applet:set_status(303)
		applet:add_header("content-type", "text/html")
		applet:add_header("location", value)
		applet:start_response()
		local response = [[
<html>
	<head>
		<meta http-equiv="refresh" content="%s">
	</head>
	<body>
		<a href="%s">%s</a>
	</body>
</html>]]

		applet:send(response:format(value, value, value))
	else
		error(applet, 404)
	end
end

Cleaning it up

After I finished the script I cleaned it up, by decorating the applet object with helper methods. For example the error function can now be called as applet:send_error.

h_app_roxy.lua
local function App(applet)
	function applet:parse_body()
		if self.headers["content-type"] == nil or #self.headers["content-type"] > 1 or self.headers["content-type"][0] ~= "application/x-www-form-urlencoded" then
			return nil
		end
		
		return decode(self:receive())
	end
	
	function applet:send_error(number)
		core.Debug("Sending error: " .. number)
		self:set_status(number)
		self:start_response()
		self:send(number)
	end
	
	function applet:send_redirect(location, code)
		code = code or 303

		core.Debug("Sending redirect: " .. code)
		applet:set_status(code)
		applet:add_header("content-type", "text/html")
		applet:add_header("location", location)
		applet:start_response()
		local response = [[
<html>
	<head>
		<meta http-equiv="refresh" content="%s">
	</head>
	<body>
		<a href="%s">%s</a>
	</body>
</html>]]
		
		self:send(response:format(location, location, location))
	end
end

The router now decorates the applet using: App(applet):

h_app_roxy.lua
local function h_app_roxy(controllers)
	return function(applet)
		App(applet)

		applet:add_header("X-Powered-By", "h-app-roxy")

		local pattern_match = false
		for i, v in ipairs(controllers) do
			if applet.path:match("^" .. v["pattern"] .. "$") then
				pattern_match = true

				if applet.method == v["method"] then
					core.Debug("Found matching controller: " .. v["method"] .. " " .. v["pattern"])
					return v["controller"](applet, applet.path:match("^" .. v["pattern"] .. "$"))
				end
			end
		end

		if pattern_match then
			applet:send_error(405)
		else
			applet:send_error(404)
		end
	end
end

Wrapping it up

The result of this journey is a re-usable application framework “h-app-roxy”. It currently comes with an easy to use request router that delegates to controller functions.

Using this framework a simple URL shortener application was built as a proof of concept. This URL shortener application can reply with about 2500 redirects per second on the authors laptop, measured using ab with a concurrency level of 150.

Abusing the Lua API of haproxy like this surfaced one MAJOR bug inside of haproxy: If Redis closed the connection, while haproxy was trying to read from it, haproxy crashed with a segmentation fault. The bug was fixed by the author of this post. The bugfix will be released with haproxy 1.8.9.

I’ve installed a live demo with a few small modifications on my server: I modified the routing to expect the /demo/prefix, I added some CSS and any generated URLs expire after 60 seconds. I don’t want to be liable for links pointing to nefarious content.