Eschewing black box API calls

I love dynamically typed languages as much as the next person. They let us make ergonomic API calls like this:

or this:

In both cases, running the snippets will return:

Unless you've worked with a statically typed language that enforces more constraints, it's hard to appreciate how incredibly convenient it is to be able to call and use an API endpoint without having to deal with types or knowing anything about its payload structure. You can treat the API response as a black box and deal with everything in runtime.

For example, Go wouldn't even allow you to do so in such a loosey-goosey way. To consume the API, you'd need to create a struct in the essence of the return payload and then unmarshal the payload into it.

Here's the complete response payload that curl -s https://dummyjson.com/products/1 | jq returns:

This is how you'd call the API endpoint in Go. I'm using the json-to-go service to generate the Go struct instead of handwriting it:

This will give us the same output as the Python and JS code snippets:

Above, we had to create a new struct type to represent the response payload, instantiate it, and unmarshal the JSON payload into the struct before we were able to process it.

Notice that we're only using 3 fields and ignoring the rest. In this case, you can get away with only including those 3 fields in the struct type, and Go will do the right thing:

While this is less work than having to emulate the whole structure of the JSON output in the struct definition, it's still not winning any medals for brevity against the Python and JS snippets.

Dynamically processing a JSON payload is nice as long as you're working on a throwaway script. Anything more, it becomes a headache since the readers won't have any idea about what the API response looks like without looking at the documentation or traces.

Also, type safety is an issue. Since the imperative workflow doesn't assume the structure of the response, you'll be surprised with a runtime error if you make an incorrect assumption about the response structure. Sure, having to write a struct is a chore, but the free documentation and the type safety are things that you don't get with the black box API calls.

Statically typed languages force you to maintain good hygiene while working with JSON payloads. Declaratively embedding the payload structure directly into the codebase is immensely beneficial; it reduces the out-of-band knowledge required to understand the code and adds type safety as a cherry on top. But how do you do that in a language like Python?

If you want to go with what's in the standard library, you can handroll a dataclass like this and project the return payload onto it:

Then just call Product.from_dict and pass the output of response.json() as before.

This way, the API response is documented in the code, and the reader won't have to depend on out-of-band information while reading the code. However, you can see that hand rolling data classes can quickly become hairy when you have a large JSON payload and need to reconcile the discrepancies between snake case and camel case variables. We had to add a custom from_dict class method to convert the camel case variables to their snake case counterparts in Python.

Also, unlike Go, you can't define a structure to represent only a portion of the whole payload in Python without adding extra code to ignore the rest of the fields that aren't relevant to you.

Pydantic shines here. It not only allows you to define a class to represent a partial payload structure, but also applies runtime validation to guarantee operational type safety. As a bonus, you can use the json-to-pydantic tool to generate pydantic classes from JSON:

You can project your response onto the data class with Product(response.json()) and get a rich object that also validates the incoming values. This will work the same way with partially defined classes:

Here's a complete example:

In the JS land, you can adopt TypeScript and zod to achieve a similar result:

I don't mind the added verbosity if it leads to better readability and type safety.

Fin!