It All Started With a Bug
Working with URLs in JavaScript and Node.js should be straightforward, but a recent bug in our project led me down a rabbit hole of subtle quirks in the URL and URLSearchParams APIs. This post will explore these quirks, how they can cause problems in your code, and what you can do to avoid them.
The Problem: URL Handling with Axios
We encountered this issue while generating URLs and adding hash signatures to them. The query parameters weren’t consistently percent-encoded, leading to unexpected behavior and wrong hash signatures.
It became clear that the interaction between URL and URLSearchParams objects required extra care.
Pitfall #1: URL.search vs. URLSearchParams.toString()
The first surprise was the difference between URL.search and URLSearchParams.toString().
Use care when using .searchParams to modify the URL because, per the WHATWG specification, the URLSearchParams object uses different rules to determine which characters to percent-encode. For instance, the URL object will not percent-encode the ASCII tilde (~) character, while URLSearchParams will always encode it.
const url = new URL("https://example.com?param=foo bar");
console.log(url.search);
console.log(url.searchParams.toString());
const myURL = new URL('https://example.org/abc?foo=~bar');
console.log(myURL.search);
myURL.searchParams.sort();
console.log(myURL.search);
In our project, we needed to explicitly reassign url.search = url.searchParams.toString() to ensure the query string was encoded consistently.
Pitfall #2: The Plus Sign Dilemma
Another gotcha is how URLSearchParams handles + characters. By default, URLSearchParams interprets + as a space, which may lead to data corruption when encoding binary data or Base64 strings.
const params = new URLSearchParams("bin=E+AXQB+A");
console.log(params.get("bin"));
One solution is to use encodeURIComponent before appending values to URLSearchParams:
params.append("bin", encodeURIComponent("E+AXQB+A"));
More details are available in the MDN documentation.
Pitfall #3: URLSearchParams.get vs. URLSearchParams.toString()
Another subtlety arises when comparing the outputs of URLSearchParams.get and URLSearchParams.toString. For example:
const params = new URLSearchParams("?key=value&key=other");
console.log(params.get("key"));
console.log(params.toString());
In multi-valued scenarios, get returns only the first value, while toString serializes all.
The Fix in Our Codebase
In our project, we resolved the issue by explicitly reassigning the search property:
url.search = url.searchParams.toString();
url.searchParams.set(
"hash",
cryptography.createSha256HmacBase64UrlSafe(url.href, SECRET_KEY ?? "")
);
This ensured that all query parameters were properly encoded before adding the hash value.
Node.js querystring module
The WHATWG URLSearchParams interface and the querystring module have a similar purpose, but the purpose of the querystring module is more general, as it allows the customization of delimiter characters (& and =). On the other hand, URLSearchParams API is designed purely for URL query strings.
querystring is more performant than URLSearchParams but is not a standardized API. Use URLSearchParams when performance is not critical or when compatibility with browser code is desirable.
When using URLSearchParams unlike querystring module, duplicate keys in the form of array values are not allowed. Arrays are stringified using array.toString(), which simply joins all array elements with commas.
const params = new URLSearchParams({
user: 'abc',
query: ['first', 'second'],
});
console.log(params.getAll('query'));
console.log(params.toString());
With querystring module, the query string 'foo=bar&abc=xyz&abc=123' is parsed into:
{
"foo": "bar",
"abc": ["xyz", "123"]
}
Takeaways
Be cautious of how URLSearchParams handles special characters (e.g. ~) and spaces. Use encodeURIComponent when necessary.
Understand the difference between URL.search, URLSearchParams.get, and URLSearchParams.toString to avoid unexpected behavior.
In Node.js use querystring module if you want to parse duplicate query parameter keys as an array.