# Pitfalls of URL and URLSearchParams in JavaScript

## It All Started With a Bug

Working with URLs in JavaScript and Node.js should be straightforward, but a recent bug in our project led me down a rabbit hole of subtle quirks in the [`URL` and `URLSearchParams` APIs](https://nodejs.org/api/url.html). This post will explore these quirks, how they can cause problems in your code, and what you can do to avoid them.

---

## The Problem: URL Handling with Axios

We encountered this issue while generating URLs and adding hash signatures to them. The query parameters weren’t consistently percent-encoded, leading to unexpected behavior and wrong hash signatures.

It became clear that the interaction between `URL` and `URLSearchParams` objects required extra care.

---

### Pitfall #1: `URL.search` vs. `URLSearchParams.toString()`

The first surprise was the difference between `URL.search` and `URLSearchParams.toString()`.

Use care when using `.searchParams` to modify the `URL` because, per the [WHATWG specification](https://url.spec.whatwg.org/), the `URLSearchParams` object uses different rules to determine which characters to percent-encode. For instance, the `URL` object will not percent-encode the ASCII tilde (`~`) character, while `URLSearchParams` will always encode it.

```typescript
// Example 1
const url = new URL("https://example.com?param=foo bar");
console.log(url.search); // prints param=foo%20bar
console.log(url.searchParams.toString()); // prints ?param=foo+bar

// Example 2
const myURL = new URL('https://example.org/abc?foo=~bar');
console.log(myURL.search);  // prints ?foo=~bar
// Modify the URL via searchParams...
myURL.searchParams.sort();
console.log(myURL.search);  // prints ?foo=%7Ebar
```

In our project, we needed to explicitly reassign `url.search = url.searchParams.toString()` to ensure the query string was encoded consistently.

---

### Pitfall #2: The Plus Sign Dilemma

Another gotcha is how `URLSearchParams` handles `+` characters. By default, `URLSearchParams` interprets `+` as a space, which may lead to data corruption when encoding binary data or Base64 strings.

```typescript
const params = new URLSearchParams("bin=E+AXQB+A");
console.log(params.get("bin")); // "E AXQB A"
```

One solution is to use `encodeURIComponent` before appending values to `URLSearchParams`:

```typescript
params.append("bin", encodeURIComponent("E+AXQB+A"));
```

More details are available in the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/URLSearchParams#preserving_plus_signs).

---

### Pitfall #3: `URLSearchParams.get` vs. `URLSearchParams.toString()`

Another subtlety arises when comparing the outputs of `URLSearchParams.get` and `URLSearchParams.toString`. For example:

```typescript
const params = new URLSearchParams("?key=value&key=other");
console.log(params.get("key")); // "value" (first occurrence)
console.log(params.toString()); // "key=value&key=other" (all occurrences serialized)
```

In multi-valued scenarios, `get` returns only the first value, while `toString` serializes all.

---

## The Fix in Our Codebase

In our project, we resolved the issue by explicitly reassigning the `search` property:

```typescript
url.search = url.searchParams.toString();
url.searchParams.set(
  "hash",
  cryptography.createSha256HmacBase64UrlSafe(url.href, SECRET_KEY ?? "")
);
```

This ensured that all query parameters were properly encoded before adding the `hash` value.

---

## Node.js `querystring` module

The WHATWG `URLSearchParams` interface and the [`querystring` module](https://nodejs.org/api/querystring.html) have a similar purpose, but the purpose of the `querystring` module is more general, as it allows the customization of delimiter characters (`&` and `=`). On the other hand, `URLSearchParams` API is designed purely for URL query strings.

`querystring` is more performant than `URLSearchParams` but is not a standardized API. Use `URLSearchParams` when performance is not critical or when compatibility with browser code is desirable.

When using `URLSearchParams` unlike `querystring` module, duplicate keys in the form of array values are not allowed. Arrays are stringified using `array.toString()`, which simply joins all array elements with commas.

```typescript
const params = new URLSearchParams({
  user: 'abc',
  query: ['first', 'second'],
});
console.log(params.getAll('query'));
// Prints [ 'first,second' ]
console.log(params.toString());
// Prints 'user=abc&query=first%2Csecond'
```

With `querystring` module, the query string `'foo=bar&abc=xyz&abc=123'` is parsed into:

```typescript
{
  "foo": "bar",
  "abc": ["xyz", "123"]
}
```

---

## Takeaways

1. Be **cautious of how** `URLSearchParams` handles special characters (e.g. `~`) and spaces. Use `encodeURIComponent` when necessary.
    
2. **Understand the difference between** `URL.search`, `URLSearchParams.get`, and `URLSearchParams.toString` to avoid unexpected behavior.
    
3. **In Node.js use** `querystring` **module if** you want to parse duplicate query parameter keys as an array.