{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/misc/eschewing-black-box-api-calls/",
  "description": "Why you should define API response structures explicitly. Compare approaches in Python, JavaScript, and Go with Pydantic, Zod, and structs.",
  "path": "/misc/eschewing-black-box-api-calls/",
  "publishedAt": "2024-01-15T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "JavaScript",
    "Go"
  ],
  "textContent": "I love dynamically typed languages as much as the next person. They let us make ergonomic\nAPI calls like this:\n\nor this:\n\nIn both cases, running the snippets will return:\n\nUnless you've worked with a statically typed language that enforces more constraints, it's\nhard to appreciate how incredibly convenient it is to be able to call and use an API\nendpoint without having to deal with types or knowing anything about its payload structure.\nYou can treat the API response as a black box and deal with everything in runtime.\n\nFor example, Go wouldn't even allow you to do so in such a loosey-goosey way. To consume the\nAPI, you'd need to create a struct in the essence of the return payload and then unmarshal\nthe payload into it.\n\nHere's the complete response payload that curl -s https://dummyjson.com/products/1 | jq\nreturns:\n\nThis is how you'd call the API endpoint in Go. I'm using the [json-to-go] service to\ngenerate the Go struct instead of handwriting it:\n\nThis will give us the same output as the Python and JS code snippets:\n\nAbove, we had to create a new struct type to represent the response payload, instantiate it,\nand unmarshal the JSON payload into the struct before we were able to process it.\n\nNotice that we're only using 3 fields and ignoring the rest. In this case, you can get away\nwith only including those 3 fields in the struct type, and Go will do the right thing:\n\nWhile this is less work than having to emulate the whole structure of the JSON output in the\nstruct definition, it's still not winning any medals for brevity against the Python and JS\nsnippets.\n\nDynamically processing a JSON payload is nice as long as you're working on a throwaway\nscript. Anything more, it becomes a headache since the readers won't have any idea about\nwhat the API response looks like without looking at the documentation or traces.\n\nAlso, type safety is an issue. Since the imperative workflow doesn't assume the structure of\nthe response, you'll be surprised with a runtime error if you make an incorrect assumption\nabout the response structure. Sure, having to write a struct is a chore, but the free\ndocumentation and the type safety are things that you don't get with the black box API\ncalls.\n\nStatically typed languages force you to maintain good hygiene while working with JSON\npayloads. Declaratively embedding the payload structure directly into the codebase is\nimmensely beneficial; it reduces the out-of-band knowledge required to understand the code\nand adds type safety as a cherry on top. But how do you do that in a language like Python?\n\nIf you want to go with what's in the standard library, you can handroll a dataclass like\nthis and project the return payload onto it:\n\nThen just call Product.from_dict and pass the output of response.json() as before.\n\nThis way, the API response is documented in the code, and the reader won't have to depend on\nout-of-band information while reading the code. However, you can see that hand rolling data\nclasses can quickly become hairy when you have a large JSON payload and need to reconcile\nthe discrepancies between snake case and camel case variables. We had to add a custom\nfrom_dict class method to convert the camel case variables to their snake case\ncounterparts in Python.\n\nAlso, unlike Go, you can't define a structure to represent only a portion of the whole\npayload in Python without adding extra code to ignore the rest of the fields that aren't\nrelevant to you.\n\n[Pydantic] shines here. It not only allows you to define a class to represent a partial\npayload structure, but also applies runtime validation to guarantee operational type safety.\nAs a bonus, you can use the [json-to-pydantic] tool to generate pydantic classes from JSON:\n\nYou can project your response onto the data class with Product(response.json()) and get\na rich object that also validates the incoming values. This will work the same way with\npartially defined classes:\n\nHere's a complete example:\n\nIn the JS land, you can adopt TypeScript and [zod] to achieve a similar result:\n\nI don't mind the added verbosity if it leads to better readability and type safety.\n\nFin!\n\n\n\n\n[json-to-go]:\n    https://mholt.github.io/json-to-go/\n\n[pydantic]:\n    https://docs.pydantic.dev/latest/\n\n[json-to-pydantic]:\n    https://jsontopydantic.com/\n\n[zod]:\n    https://github.com/colinhacks/zod",
  "title": "Eschewing black box API calls"
}