Yet another opinion on LLMs · Hasufell's blog

lortabac:

These tools (especially the most advanced models) can be incredibly helpful, not just for mechanical tasks. They can also help with reasoning, find subtle bugs, analyze data to spot patterns and correlations, brainstorm…

Yesterday I was investigating a bug in the GHC bindist configure and ran across an m4 macro:

# FPTOOLS_OVERRIDE_PLATFORM_FROM_BOOTSTRAP(platform,Platform)
# ----------------------------------
# Per the comment in FPTOOLS_OVERRIDE_PLATFORM_FROM_BOOTSTRAP's body, we
# need to sometimes replace inferred platforms with the bootstrap
# compiler's target platform.
AC_DEFUN([FPTOOLS_OVERRIDE_PLATFORM_FROM_BOOTSTRAP],
[
    if test "$bootstrap_target" != ""
    then
        $1=$bootstrap_target
        echo "$1 platform inferred as: [$]$1"
    else
        echo "Can't work out $1 platform"
        exit 1
    fi

    $2[Arch]=`echo "[$]$1" | sed 's/-.*//'`
    $2[Vendor]=`echo "[$]$1" | sed -e 's/.*-\(.*\)-.*/\1/'`
    $2[OS]=`echo "[$]$1" | sed 's/.*-//'`
])

And the call site:

    if test "$host_alias" = ""
    then
        FPTOOLS_OVERRIDE_PLATFORM_FROM_BOOTSTRAP([host], [Host])
    else
        GHC_CONVERT_PLATFORM_PARTS([host], [Host])
    fi

So if you don’t pass --host to configure it will infer it from ghc +RTS --info. But then it doesn’t synchronize it with the autoconf variables host_os, host_cpu and host_vendor, leading to subtle issues.

My first version of a patch had a bug:

--- a/m4/fptools_set_platform_vars.m4
+++ b/m4/fptools_set_platform_vars.m4
@@ -89,6 +89,13 @@ AC_DEFUN([FPTOOLS_OVERRIDE_PLATFORM_FROM_BOOTSTRAP],
     $2[Arch]=`echo "[$]$1" | sed 's/-.*//'`
     $2[Vendor]=`echo "[$]$1" | sed -e 's/.*-\(.*\)-.*/\1/'`
     $2[OS]=`echo "[$]$1" | sed 's/.*-//'`
+
+       # synchronize with the autotools variables, e.g.
+       # target_cpu = TargetPlatform
+       # etc.
+       $1[_cpu]=$2[Arch]
+       $1[_vendor]=$2[Vendor]
+       $1[_os]=$2[OS]
 ])

 # FPTOOLS_SET_PLATFORM_VARS(platform,Platform)

Kudos if you can spot it right away. I couldn’t, so I went ahead and asked Gemini what the problem with this patch is.

It kept telling me that $1[_cpu] doesn’t work, but $2[Arch] is fine on the left-hand side of these assignments, suggesting it’s actually the underscore that trips up the parser or whatever.

Here’s an excerpt from the prompt: gemini.md · GitHub

This was utter nonsense. It also kept suggesting obscure solutions that involve eval and other things. It clearly doesn’t know basic m4 syntax and has no way of actually reasoning about it.

The problem with the patch was simply that it was missing an escaped dollar sign, so instead of $1[_cpu]=$2[Arch] I needed $1[_cpu]=[$]$2[Arch]. My brain was on autopilot, because I hate m4 and didn’t want to think about it. Then I wasted 15 minutes of my time arguing with Gemini and demanding an explanation why it thinks the underscore is the issue. It was extremely confident about it.

lortabac:

The truth is, if you want to get useful results you need to sweat.

Yes, people keep telling me that I simply don’t know how to use LLMs correctly.

My guess is that most users would just have copy pasted the obscure code from Gemini and moved on instead of questioning why it came up with said code. And maybe it would have even worked.

No, I don’t think there’s any depth about interacting with LLMs.

lortabac:

Your post dismisses all these capabilities a little too hastily.

I don’t think so, because I provide criteria for what is “good use of LLMs”:

1. when you can trivially verify the output, or
2. when you have superior domain knowledge, or
3. when you don’t need precise answers

In my experience though, this is not how people use them, because this indeed takes effort. And then the gains are not as big anymore.