Blogs by Luciano Nooijen

Creating and validating JWTs with public/private keys (JSON Web Key Set, JWKS)

Wed, 16 Apr 2025 00:00:00 +0300

This article discusses an implementation of JWT generation in Go, and validation in Go and Typescript. The library used for this, jose, has implementations for many other programming languages, so the principles can be applied in nearly all commonly used programming languages.

JSON Web Token¹ (JWT for short) is a very commonly used way to create and validate authentication tokens on the web. This article is not a tutorial of JWT basics, there are many resources available for that online already.

Most often, when generating JWTs, a secret value is used to create the third part of the JWT, the signature. This approach is great when there is a single service that generates and validates tokens. In some cases though, you want to sign a token on one service, and have it validated by another service. This could be achieved by sharing the signing secret, but this is not an optimal solution in terms of security, you would rather sign a JWT on one service and then validate it on another service, in a way that doesn’t require you to have access to the secret value.

This can be achieved by using private/public key encryption. You can sign a JWT with a private key, and in the header include the link to where the public key can be found to validate the signature. This will allow other services to validate a JWT, without requiring access to the actual secret value to sign a key. To make this more concrete: this system is used by for example Auth0² to sign keys to then allow you to validate these tokens without needing access to the signing secret. This system is also used by AWS Cognito³.

I recently integrated this into Capsa.gg . The web panel has middleware to validate JWTs, which will redirect a user to log in if the key is invalid. When I started implementing this, I was not able to find a lot of resources on this unfortunately and it took quite some trial-and-error to complete. Yet in the end, implementing JWTs with private key signing and public key validation was quite straight-forward. Hence why I’m sharing my findings here, in the hope it will be useful to others.

This article will implement JWT generation with a private key and gives an example of validating with the private key. Some code is omitted, a working example project can be found on Github: lucianonooijen/jwt-public-private-key-demo . This also contains some unit tests for validation, which are not included in this article.

❗

Security disclaimer

Please don’t follow this guide as if it’s gospel, make sure you know what you are doing. This article and the example project are meant as starting points, to be adopted according to specific needs. Never blindly trust code on the internet, especially when it comes to security. The footnotes of this article contain trusted sources that can be used for further reading.

JSON Web Key

When generating a JWT, the first part of the token, the header, contains information about how the key is signed. The most important field for our use-case, is the algorithm field alg.

For example, when signing a token with just a simple secret, the header contents are

{
 "alg": "HS256",
 "typ": "JWT"
}

indicating that the algorithm used is “HMAC using SHA-256”⁴. This is the easiest way to sign and verify JWTs, used in many projects. You need a secret to sign the token, and use the same token to validate it.

Now let’s look at the header of a JWT that has been signed with the public/private key setup that this article discusses:

{
 "alg": "RS256",
 "jku": "http://localhost:4000/.well-known/jwks.json",
 "typ": "JWT"
}

The algorithm field here now has a different value, RS256, indicating the digital signature algorithm used is “RSASSA-PKCS1-v1_5 using SHA-256”⁵.

This looks like a cat walked across the keyboard, but we can break this down. RSA indicates that we are using RSA, so a public/private key setup, SSA indicates “With Appendix”, meaning the signature is separate from the message, which makes sense with how JWTs work. PKCS1 refers to the “Public-Key Cryptography Standards #1”⁶, which defines how RSA should be used. v1_5 indicates the padding scheme, which is older, but used and supported in JWTs.

A new field in the header, is the jku field. This is the JWK Set URL⁷, often being <service_url>/.well-known/jwks.json⁸. The URL here should resolve to the public key data when making a GET request to this URL over TLS⁹. The value of this field should not be blindly trusted at runtime unless it matches an expected value, or there is the risk of a key injection attack. When fetching this URL, there is a specific format how the data should be encoded¹⁰, which is called the JSON Web Key Set, or JWKS. “Set” here meaning a collection of multiple JSON Web Keys (singular: JWK). We will discuss this a bit more later. The important takeaway here is: this URL contains the public key data of all signing keys that should be considered valid.

A small note to add: for the example project, we are not fully following the RFC, as we fetch the JWKS without TLS. When implementing a JWKS endpoint, it should always be to over TLS to follow the RFC⁷.

To summarize: when implementing public/private key signing for JWTs, the JWT should contain the correct algorithm, as well as a URL where valid keys can be retrieved for validating the tokens. There is a specific format on how the public keys must be made available.

Public/private key generation

So, let’s start implementing JWT signing with public/private keys. The first thing we need, is to actually generate the keys. We need to generate keys with at least 2048 bits¹¹.

We don’t want to generate a key, keep it in memory and use that right away. The keys should be reused if the application starts up. You can generate the key in any way you like, this is an example of how to do this in Go. Let’s assume we want to create a private key, then extract the public key from that, encode that as strings that we will return to the calling function, which will write the keys to disk.

package token

import (
 "crypto/rand"
 "crypto/rsa"
 "crypto/x509"
 "encoding/pem"
 "fmt"
)

// RsaKeySet is a set of generated public and private keys.
type RsaKeySet struct {
 PrivateKey []byte
 PublicKey []byte
}

// GenerateRsaKeySet generates a private/public key set for signing JWKs.
func GenerateRsaKeySet() (*RsaKeySet, error) {
 privateKey, err := rsa.GenerateKey(rand.Reader, 4096)
 if err != nil {
 return nil, fmt.Errorf("error generating private key: %w", err)
 }

 return EncodePrivateKeyToBytes(privateKey)
}

// EncodePrivateKeyToBytes takes in a *rsa.PrivateKey and encodes this to the private and public PEM key bytes.
func EncodePrivateKeyToBytes(privateKey *rsa.PrivateKey) (*RsaKeySet, error) {
 privateKeyBytes := encodePrivateKeyToPEM(privateKey)

 publicKeyBytes, err := encodePublicKeyToPem(privateKey)
 if err != nil {
 return nil, fmt.Errorf("error generating public key: %w", err)
 }

 keyData := RsaKeySet{
 PrivateKey: privateKeyBytes,
 PublicKey: publicKeyBytes,
 }

 return &keyData, nil
}

func encodePrivateKeyToPEM(privateKey *rsa.PrivateKey) []byte {
 // Get PKCS #1, ASN.1 DER format
 privateKeyContents := x509.MarshalPKCS1PrivateKey(privateKey)

 // pem.Block
 privateBlock := pem.Block{
 Type: "RSA PRIVATE KEY",
 Headers: nil,
 Bytes: privateKeyContents,
 }

 // Private key in PEM format
 privatePEM := pem.EncodeToMemory(&privateBlock)

 return privatePEM
}

func encodePublicKeyToPem(privateKey *rsa.PrivateKey) ([]byte, error) {
 // Extract the public key
 publicKey := &privateKey.PublicKey

 // Marshal the public key to PKIX, ASN.1 DER form
 publicKeyBytes, err := x509.MarshalPKIXPublicKey(publicKey)
 if err != nil {
 return nil, fmt.Errorf("error marshaling public key: %w", err)
 }

 // Create the PEM block
 publicKeyPEMBlock := pem.Block{
 Type: "PUBLIC KEY",
 Bytes: publicKeyBytes,
 }

 // Encode the PEM block to []byte
 publicKeyPEM := pem.EncodeToMemory(&publicKeyPEMBlock)

 return publicKeyPEM, nil
}

Security disclaimer: keys must be handled with care. You need to use different keys for different environments and have proper practices for handling secrets. Please make sure you handle key generation and storage with great care.

Accessing the public and private key

So we have generated a private and public key and persisted the contents to disk. Let’s now work on the actual JWT logic that uses these keys. In practice, we don’t need to load the public key from disk, only the private key, which we will use to get the public key values.

So let’s write two helper functions: one that will load the private key from disk, and another that will decode a base64 string that is passed into it, for example if the private key value is set as an environment variable, base64 encoded.

package token

import (
 "crypto/rsa"
 "crypto/x509"
 "encoding/base64"
 "encoding/pem"
 "errors"
 "fmt"
 "os"
)

const blockTypeRsaPrivateKey = "RSA PRIVATE KEY"

var (
 // ErrorPemPrivateKeyDecoding indicates that the decoding of the private-key containing PEM block failed.
 ErrorPemPrivateKeyDecoding = errors.New("failed to decode PEM block containing private key")
)

// LoadPrivateKeyFromPath reads the private key from a file and returns a rsa.PrivateKey struct.
func LoadPrivateKeyFromPath(path string) (*rsa.PrivateKey, error) {
 // Read the file
 keyBytes, err := os.ReadFile(path) //nolint:gosec // "G304: Potential file inclusion via variable", this path is set in the config.yml and should never contain user input
 if err != nil {
 return nil, fmt.Errorf("failed to read private key file: %w", err)
 }

 // Decode the PEM block
 block, _ := pem.Decode(keyBytes)
 if block == nil || block.Type != blockTypeRsaPrivateKey {
 return nil, ErrorPemPrivateKeyDecoding
 }

 // Parse the private key
 privateKey, err := x509.ParsePKCS1PrivateKey(block.Bytes)
 if err != nil {
 return nil, fmt.Errorf("failed to parse private key: %w", err)
 }

 return privateKey, nil
}

// LoadPrivateKeyFromBase64String reads the private key from a base64 string and returns a rsa.PrivateKey struct.
func LoadPrivateKeyFromBase64String(b64 string) (*rsa.PrivateKey, error) {
 keyBytes, err := base64.StdEncoding.DecodeString(b64)
 if err != nil {
 return nil, fmt.Errorf("failed to decode private key base64 string: %w", err)
 }

 // Decode the PEM block
 block, _ := pem.Decode(keyBytes)
 if block == nil || block.Type != blockTypeRsaPrivateKey {
 return nil, ErrorPemPrivateKeyDecoding
 }

 // Parse the private key
 privateKey, err := x509.ParsePKCS1PrivateKey(block.Bytes)
 if err != nil {
 return nil, fmt.Errorf("failed to parse private key: %w", err)
 }

 return privateKey, nil
}

We now have our *rsa.PrivateKey instance again that we can use to sign JWTs with.

Signing JWTs

To sign the JWTs, we will be using go-jose¹². Let’s create a struct called Token, which holds the data we need, and exposes methods to sign and validate tokens:

import (
 "crypto/rsa"
 "errors"
 "fmt"

 "github.com/go-jose/go-jose/v4"
)

type Token struct {
 privateKey *rsa.PrivateKey
 jwk *jose.JSONWebKey
 signer *jose.Signer
}

With our Token struct defined, let’s create a function called New in our package, that will accept a private key and returns an initialized Token instance:

const (
 keyID = "jwt-demo-server-jwk"
 algorithm = jose.RS256
)

// New returns a Token instance after validating the *rsa.PrivateKey.
func New(pk *rsa.PrivateKey) (*Token, error) {
 if pk == nil {
 return nil, errors.New("private key argument is required")
 }

 if err := pk.Validate(); err != nil {
 return nil, fmt.Errorf("error validating private key: %w", err)
 }

 jwk := jose.JSONWebKey{
 Key: pk,
 Use: "sig",
 Algorithm: string(algorithm),
 KeyID: keyID,
 }

 signerOptions := (&jose.SignerOptions{}).
 WithHeader("alg", algorithm).
 WithHeader("typ", "JWT").
 WithHeader("jku", "http://localhost:4000/.well-known/jwks.json")

 jwtSigner, err := jose.NewSigner(jose.SigningKey{
 Algorithm: algorithm,
 Key: pk,
 }, signerOptions)

 if err != nil {
 return nil, fmt.Errorf("error creating jwt signer: %w", err)
 }

 jwkInstance := Token{
 privateKey: pk,
 jwk: &jwk,
 signer: &jwtSigner,
 }

 return &jwkInstance, nil
}

A few important notes here: the keyID is an identifier for which type of key we are dealing with. If you have two token types: a user and an admin, you want to use different key IDs, which other services should validate if it’s the right keyID. The jku field is now hardcoded in this example, you will most likely need to add a method that uses the application config to generate the correct URL, which is using https, not http like in this example. In the jose.JSONWebKey, we set Use: "sig", because we use this key for signing, not for encrypting the JWT¹³, that is beyond the scope of this article.

With our new Token instance, let’s add a few methods that allows us to actually generate and validate JWTs. For our example application, we will only have a single token type. When dealing with multiple token types, you will most likely have to generate multiple functions for generation and validation, for each token type.

Let’s generate a JWT:

const (
 Audience = "example"
 Issuer = "jwt-demo-server"
)


func (t *Token) GenerateJwt(subject, jwtId, name, role string) (string, error) {
 now := time.Now()
 expiryHours := time.Duration(48) * time.Hour

 claims := JwtClaims{
 Issuer: Issuer,
 Subject: subject,
 Audience: Audience,
 Expiry: now.Add(expiryHours).Unix(),
 NotBefore: now.Unix(),
 IssuedAt: now.Unix(),
 JwtID: jwtId,
 Name: name,
 Role: role,
 }

 tok, err := t.generateTokenForClaims(claims)
 if err != nil {
 return "", fmt.Errorf("error signing token: %w", err)
 }

 log.Printf("generated client jwt with subject: %s\n", subject)

 return tok, nil
}

// generateTokenForClaims is a function that will sign the JwtClaims passed in.
// WARNING: This method should only be called in public wrapper functions and not exposed directly.
// For generating tokens in production code, always use the audience-specific methods.
func (t *Token) generateTokenForClaims(claims JwtClaims) (string, error) { //nolint:gocritic // Jose needs val, not ref
 return jwt.Signed(*t.signer).Claims(claims).Serialize()
}

The scary looking second function performs the logic to actually generate the JWT string with the claims passed in. Discussing the JWT claims is beyond the scope of this article, these are explained in the RFC¹. An example can be found in the example project . The note “For generating tokens in production code, always use the audience-specific methods.” refers to using wrapper functions to generate keys with the correct data.

Validating JWTs using the private key

Of course we also want to be able to validate the JWT on the service that generated the key. Let’s add that before we implement the validation on an external service:

var (
 // ErrorJwtParsing indicates that the signature of the JWT is not valid.
 ErrorJwtParsing = errors.New("parsing JWT failed")

 // ErrorJwtValidation indicates that the token claims could not be validated.
 ErrorJwtValidation = errors.New("claim validation for JWT failed")

 // ErrorJwtConversion indicates that the token claims to struct conversion failed.
 ErrorJwtConversion = errors.New("claim conversion for JWT failed")
)

func (t *Token) ValidateJwt(token string) (*JwtClaims, error) {
 parsedToken, err := jwt.ParseSigned(token, []jose.SignatureAlgorithm{algorithm})
 if err != nil {
 log.Printf("error parsing signed token: %s\n", err)

 return nil, ErrorJwtParsing
 }

 claims := Claims{}

 // Note: dereference here is very important!
 // The jose code checks for *rsa.PublicKey specifically, and does not accept rsa.PublicKey
 err = parsedToken.Claims(&t.privateKey.PublicKey, &claims)
 if err != nil {
 log.Printf("error validating token claims: %s\n", err)

 return nil, ErrorJwtValidation
 }

 c, err := claims.jwtClaims()
 if err != nil {
 log.Printf("error converting token claims: %s\n", err)

 return nil, ErrorJwtConversion
 }

 err = validateJwtClaims(c)
 if err != nil {
 log.Printf("error validating token claims: %s\n", err)

 return nil, err
 }

 return c, nil
}

This code assumes a type Claims map[string]any definition. On this type, we add a method jwtClaims that converts this map to our own custom claims type, which was mentioned before. For inspiration, you can check this example . We also call our own validateJwtClaims method, which is beyond the scope of this article, but it validates a bunch of fields, you can draw inspiration from this example . The validation required depends on your application. You should also validate that the algorithm and key id match the expected values, or reject the token if they don’t.

Remember that when signing tokens with RS256 (public/private key setup), that we should include a URL where the JWKS can be retrieved? Let’s add that now.

In our Token instance, we want to add a method that allows us to retrieve the JSON representation of the public key. Getting the JSON representation of the public key is very straight-forward:

// GetPublicKey takes the public part of the private key and marshals this to JSON for the .well-known/jwks.json endpoint.
func (t *Token) GetPublicKey() ([]byte, error) {
 // SECURITY NOTE: NEVER REMOVE `.Public()` FROM THIS CODE
 // Otherwise, the private key will be shared and the whole token system will be compromised!
 return t.jwk.Public().MarshalJSON()
}

Make sure you use jwk.Public()! If you do not include the Public() part in here, you will expose the full private key, meaning the whole token system will be compromised!

We can add a simple handler to return this key like this, assuming we have our handlers in a handlers struct which has an instance of Token in the token struct field:

func (h *handlers) Jwk(c *gin.Context) {
 pubKey, err := h.token.GetPublicKey()
 if err != nil {
 // handle error
 return
 }

 c.Header("content-type", "application/json")

 // Manually writing JSON for compatibility with the []byte from pubKey
 c.Writer.WriteString(`{"keys":[`) //nolint:errcheck,gosec // This is fine
 c.Writer.Write(pubKey) //nolint:errcheck,gosec // This is fine
 c.Writer.WriteString(`]}`) //nolint:errcheck,gosec // This is fine

 c.Status(http.StatusOK)
}

The proper JSON marshalling is left as an exercise for the reader.

Inspecting the JWT

Let’s add a super simple API endpoint GET /jwt that generates a JWT with some hardcoded values. This of course is just for demo purposes and not secure to be used in production. In production, add authentication checks for users to log in and use the proper JWT claim values.

Example handler for testing:

func (h *handlers) GetJwt(c *gin.Context) {
 // Never do this in production, just for demo!
 jwt, err := h.token.GenerateJwt("42", "1337", "John Doe", "Example")
 if err != nil {
 // Handle error
 return
 }

 // Validate the token passes our own validation, plus get the claims so we can return the expiry time
 claims, err := h.token.ValidateJwt(jwt)
 if err != nil {
 // Handle error
 return
 }

 c.JSON(http.StatusOK, struct {
 Token string `json:"token"`
 Expiry int64 `json:"expiry"`
 }{Token: jwt, Expiry: claims.Expiry}) // You should do this properly
}

So to test, let’s run the server and run curl http://localhost:4000/jwt | jq '.token'¹⁴ to generate a token, and parse the JWT string from the response. Let’s copy the value into jwt.io . We can see that the header values and claims are set correctly.

Validating the JWT in NextJS

So now we get to the exciting part, validating the JWT in a separate service. Let’s initialize an empty NextJS project and add some JWT validation middleware:

middleware.ts:

import {
 deleteJwtCookie,
 getJwtCookieFromRequest,
} from "@/data/jwt/cookiesServer";
import JwtValidator from "@/server/jwt";
import { type NextRequest, NextResponse } from "next/server";

// This middleware checks if users are logged in.
// This is not implemented for security reasons, as that is done on the server.
// It simply serves as a way to make sure users don't get 4xx errors.
export async function middleware(req: NextRequest) {
 const { pathname, search, origin, basePath } = req.nextUrl;
 const path = `${basePath}${pathname === "/" ? "" : pathname}${search}`;
 const logBase = `[middleware][${req.method} ${path}]:`;

 const token = await getJwtCookieFromRequest(req);
 const isAuthenticated = token ? await JwtValidator.ValidateJwt(token) : false;
 const isAuthRoute = req.nextUrl.pathname.startsWith("/auth");

 console.log(
 logBase,
 "received request, isAuthenticated",
 isAuthenticated,
 "| isAuthRoute",
 isAuthRoute,
 );

 // Logged-in users accessing login routes should be redirected to the homepage
 if (isAuthenticated && isAuthRoute) {
 console.log(
 logBase,
 "redirecting logged in user from auth route to",
 req.url,
 );

 return NextResponse.redirect(new URL("/", req.url));
 }

 // Send users who are not logged in and request non-auth pages to the login page
 if (!isAuthenticated && !isAuthRoute) {
 await deleteJwtCookie(req);

 const signInUrl = new URL(`${basePath}/auth/login`, origin);

 if (path !== "") {
 signInUrl.searchParams.set("redirect", path);
 }

 console.log(
 logBase,
 "redirecting non logged in user to",
 signInUrl.toString(),
 );

 return NextResponse.redirect(signInUrl);
 }

 console.log(logBase, "continue executing request");

 return NextResponse.next();
}

export const config = {
 matcher: [
 /*
 * Match all request paths except for the ones starting with:
 * - api (API routes)
 * - _next/static (static files)
 * - _next/image (image optimization files)
 * - favicon.ico, sitemap.xml, robots.txt (metadata files)
 */
 "/((?!api|_next/static|_next/image|favicon.ico|sitemap.xml|robots.txt).*)",
 ],
};

The actual JwtValidator implementation will be specific for each application and should take care to properly validate tokens. The implementation for the demo project can be found here . For validation, you should think of checking the algorithm, expected audience, issuer and timestamps.

It is good practice to set the JWKS endpoint in your application config, so that you only fetch keys from that location and reject keys that have a different jku field in the header. Don’t trust the jku field from the header and start fetching the key from there, in case a malicious actor set up their own service to sign keys. Security here is very important, so take good care to minimize the possible attack vectors.

Seeing this in action

The example project for this article can be found here on GitHub.

For a more complete example with different types of keys, you can take a look at Capsa’s token package and at Capsa’s NextJS middleware .

❗

Disclaimer

Never blindly trust code on the internet, especially when it comes to security. Use this article as a starting point but please do your own research and familiarize yourself with the relevant RFCs when implementing this for a production application.

RFC7519 ↩︎ ↩︎
Auth0 JSON Web Key Sets ↩︎
AWS Cognito Token Validation ↩︎
RFC7518: HMAC with SHA-2 Functions ↩︎
RFC7519: Digital Signature with RSASSA-PKCS1-v1_5 ↩︎
RFC8017 ↩︎
RFC7515: “jku” (JWK Set URL) Header Parameter ↩︎ ↩︎
This is not a strict requirement for our use-case here, but comes from the conventions for OAuth2.0 Authorization Server Metadata. This is also the endpoint used by Auth0 and AWS Cognito. See RFC8414: Authorization Server Metadata for more info. ↩︎
The RFC marks this field as optional but for our use-case we want to include this field. ↩︎
RFC7517: JSON Web Key (JWK) Format ↩︎
RFC7518: Digital Signature with RSASSA-PKCS1-v1_5 ↩︎
go-jose/go-jose ↩︎
RFC7517: “use” (Public Key Use) Parameter ↩︎
jq is a very helpful tool for working with JSON data on the command line, but how to use it are beyond the scope of this article. ↩︎

Why I stopped using AI code editors

Tue, 01 Apr 2025 00:00:00 +0300

TL;DR: I chose to make using AI a manual action, because I felt the slow loss of competence over time when I relied on it, and I recommend everyone to be cautious with making AI a key part of their workflow.

In late 2022, I used AI tools for the first time, even before the first version of ChatGPT. In 2023, I started using AI-based tools in my development workflow. Initially, I was super impressed with the capabilities of these LLMs. The fact that I could just copy and paste obscure compiler errors along with the C++ source code, and be told where the error is caused felt like magic.

Once GitHub Copilot started becoming more and more powerful, I started using it more and more. I used various other LLM integrations right in my editor. Using AI was part of my workflow.

In late 2024 I removed all LLM integrations from my code editors. I still use LLMs occasionally and I do think AI can be used in a way that is very beneficial for many programmers. So then why don’t I use AI-powered code editing tools?

Tesla FSD

From 2019 to 2021 I drove a Tesla. Though I would never make the same purchase again, not for political reasons, just because the cars are quite low quality, very overpriced and a hell to repair or maintain.

When I got my Tesla, I started using the Full Self-Driving (FSD) anytime I could. It felt great to just put the car on FSD on the highway and zone out a bit. Switching lanes was as simple as hitting the turn signal, and the car would switch lanes. Driving for me was just getting to the highway, turning on FSD, telling the car to switch lanes every now and then, and listen to music/podcasts while zoning out.

If you drive a car often, you’ll know that when you’re driving on the highway, everything sort of happens automatically. Keeping your car in the lane at the right speed becomes a passive action, it does not require the type of focus that for example reading a book requires, it’s the type of focus that walking requires, it happens in the background of your mind.

In the period from 2019 to 2021 I exclusively drove my Tesla for longer rides. After 2021, I went back to driving regular cars and making this switch was definitely not what I expected. Driving on the highway required my full attention for the first month or so, I had to re-learn keeping the car in the middle of the lane without thinking about it.

Being reliant on Tesla’s FSD took away my own ability to go into autopilot.

My experience with AI code editors

Working with AI-powered code editors was somewhat similar. Initially, I felt that I completed work a lot faster when assisted by AI. The work I was doing most of the time was not super complex, and AI felt like putting my Tesla on FSD, I could just guide the machine to do my work for me.

In my free time, I started working on a side project on my personal account on my work device. On this account, I did not have access to Copilot and my other cool, fancy AI tools. This is when using AI started to feel very similar to my Tesla FSD story.

I felt less competent at doing what was quite basic software development than a year or so before. All of a sudden, it made it very clear to me how reliant I had become on AI tools. Anytime I defined a function, I paused in my editor to wait until the AI tools would write the implementation for me. It took some effort to remember what the syntax was to write unit tests by hand.

With my work, AI started to become less useful over time as well. Not only did it take out the fun for me, but I started to feel a bit insecure about making some implementation decisions myself. Outsourcing the decisions to the AI seemed a lot easier. But sometimes, the AI couldn’t figure things out, even with the best prompts. It was quite clear that because I did not practice the basics often, I was less capable with the harder parts as well.

The loss of Fingerspitzengefühl

Fingerspitzengefühl [ˈfɪŋɐˌʃpɪtsənɡəˌfyːl] is a German term, literally meaning “finger tips feeling” and meaning intuitive flair or instinct, which has been adopted by the English language as a loanword. It describes a great situational awareness, and the ability to respond most appropriately and tactfully. ¹

Defining seniority is a very tough thing. Though in my opinion a lot of being a “senior” is in soft-skills, when it comes to the technical hard-skills, a lot comes down to Fingerspitzengefühl. The longer you work with a language, framework or codebase, the more you develop this kind of intuition of what the correct approach is. The gut feeling of “something feels off” slowly turns into a feeling of “this is what we should do”.

This developed intuition is not just on an architectural level. A big component is in the lower level details, when to use pointers (or what type of pointers), whether to use asserts or checks, what to pick from the standard library when multiple options are available (though senior C++ programmers still can’t seem to agree on this).

This intuition is what I was slowly losing when relying on AI tools a lot. And this is coming from a lead developer. When I see a lot of hype about vibe coding, I can’t help but think: how do you exactly expect to vibe code your way to senior? Where will you get the skills from to maintain and extend the vibe-coded codebase when the AI tools are down, or have become too expensive?

Even with larger context windows, more computing power, reasoning models or agents, there will be things that AI won’t be able to do. Over time, the AI tools will be more and more powerful, sure. But when you receive a Slack message that “the website works fine, but the app is down in production; I tried it locally and there it works fine, nothing in Sentry either”, good luck getting an AI agent to fix this for you. Maybe it can, maybe it can’t. And when an AI agent can’t figure it out, will your reply be “sorry, Cursor doesn’t get it, will prompt more tomorrow”?

You can get by without these tools

Sometimes it feels like you have to use AI or be out of a job in 6 months. We’ve been hearing the “3-6 months from now”-story for over two years at this point. I stopped trusting CEO promises about functionality “3-6 months from now” years ago. When I got my Tesla in 2019, I paid €6400 for functionality that was supposed to arrive in “3-6 months from now”, and the functionality is still not present the way it was promised over 5 years ago.

Right now, it is unlikely that letting AI do your coding will work for projects larger than a university project. When working on legacy systems or larger projects in enterprises or when you need to work with and consult a lot of dependency internals (like I do with Unreal Engine), AI tools will often not be able to make things work. When you need to work with internal DSLs, tools or frameworks, good luck getting LLMs to generate useful output. For some industries, you can’t even use AI tools at all for a multitude of reasons.

For some things you really should not want to rely on AI. When implementing authentication systems like JWT² signing or RBAC³, adding “and it should be secure” to the prompt won’t make it secure if it’s been trained on GitHub code that had CVEs⁴. When it comes to security, you should be the person who is responsible and understands this fully. Critical systems should be written and reviewed by humans, if we are heading to a situation where one AI agent writes the code, another reviews the autogenerated PR and then another AI agent deploys the code, we will see a huge spike of security issues soon.

Where I draw the line

I still use AI, sometimes. I think it can be a great tool, when used wisely. I draw the line at integration. I keep AI fully separate from my code editor. All of the context, I add manually. I intentionally keep the effort required quite high, so it disincentivizes me.

Examples where I use AI for work include “convert these Go tests in structs to tests in a map”, “convert this calculation to SIMD”, or “when the content type is application/zlib, decode the body”⁵. I have set up some custom instructions to only give me the code that has changed, and give me instructions for adding it. This way, I am still the one making the changes in the codebase. Just approving a Git diff is not enough, I want to manually add the code myself, only then do I feel confident to sign off on it and take responsibility for it.

Another great use case for AI is learning. I often have questions that are quite uncommon, as I have a few very niche interests. Turns out, adding netcode to a custom game engine using ECS doesn’t have a lot of learning resources. What has worked for me, is asking AI to explain pieces of code, like “explain this assembly code”, “explain what this shader does”, “which books go in-depth about resolving client/server desyncs in game engines”. The AI seems to struggle with these sometimes, I’m getting mixed results, but the results are still much better than search engines. I will even use it for this article, though not for writing content, but for checking⁶.

Another benefit of using AI this way is the cost. No unnecessary API calls, manually managed contexts and more control over the LLM settings. I use a desktop application with a bunch of different LLMs hooked up to it. I have used it daily for the last 3 months or so, and in total, I have consumed around $4 in credits.

I do want to add that with some things I am more strict. On my personal website, I don’t want any AI-generated content, whether that’s text or images. I don’t like AI generated images or ‘art’ personally for various reasons and I think AI-generated text lacks character, it feels very flat and boring. When something is created by humans, it to me has more value than when it’s created by AI.

Doing what you love

It is also worth noting that there are more things to think about than efficiency and productivity. It’s also about doing what you love. If you love coding, keep doing it yourself, even if a computer might be better at it.

In 1997, Deep Blue won the chess match against the then world chess champion Garry Kasparov⁷, yet people still play chess. When it comes to programming, I’d say that I program for the same reason that people still play chess⁸. Though chess and software development are very different, with chess being much more limited in scope, I think it is good to keep in mind that sometimes, we can do things just to enjoy them.

My advice to new programmers

Don’t become a forever junior who lets AI do all their work. If you want to become a programmer, learn to program yourself. Be curious, put in the time and effort to learn how things really work, and how things work in the layer below that. It really pays off. Learning how everything works under the hood and using that is amazing, just keep learning, don’t be a prompt engineer (if you can even call that engineering). Believe me, it’s more fun to be competent⁹.

Even though AI might be smarter than you, never blindly trust the AI output. Don’t build your whole workflow around it. Sometimes try to work without it for a few days. The better at programming you are, the more AI will get in your way for the more complex work.

If you learn to code now, keep building your skills instead of letting AI do all the heavy lifting, you’ll be capable of fixing the messes that vibe coding is now creating. I don’t want to sound elitist, but if you don’t want to learn to go beyond vibe coding, maybe coding isn’t for you. Because positions where all work can be done by vibe coding are the ones that will be eliminated first when AI becomes more powerful.

And remember: if you cannot code without AI, you cannot code.

Conclusion

When you are using AI, you are sacrificing knowledge for speed. Sometimes it’s worth making this trade-off. Though it is important to remember that even the best athletes in the world are still doing their basic drills for a reason. The same applies to software development: you need to practice the basics, to be able to do the advanced work. You need to keep your axe sharp.

We are still a long way out from AI taking over our jobs. A lot of companies are creating FOMO¹⁰ as a sales tactic to get more customers, to show traction to their investors, to get another round of funding, to generate the next model that will definitely revolutionize everything.

AI is a tool, it is not good or bad in itself, it’s what you do with it. I do think it can be a great tool, as long as you are not reliant on it for your workflow. Make sure you can still work effectively without it, make sure you don’t push code to production that you don’t fully understand and don’t think of AI as a replacement for your own thinking. Stay curious, keep learning.

Source: Wikipedia ↩︎
JSON Web Tokens, or JWTs are a common way to generate authentication tokens, among other uses ↩︎
Role-based access control (RBAC) is a mechanism to restrict system access by setting permissions and privileges ↩︎
Common Vulnerabilities and Exposures (CVE) is a program used to identify, define and catalog publicly disclosed cybersecurity vulnerabilities, cve.org ↩︎
The specific contents here don’t matter that much, they are just examples of what I use AI for ↩︎
The prompt I’ve used for this article: “I want you to proofread an article I have written. I want you to give me feedback on incorrect grammar or broken sentences, using UK grammar. Do not comment on sentences that should be broken up or things that could be improved just slightly, only real errors. Do not return modified sentences, but point out where the issue is, under which paragraph, in which sentence and what the mistake is. I will make the required changes myself”. It came back with a few typos, like “form” that should be “from”, “eb” that should be “be”. ↩︎
Source: IBM History ↩︎
Source: Tsoding on YouTube ↩︎
Source: DHH in an interview on YouTube ↩︎
Fear of missing out ↩︎

Announcing Capsa, an Unreal Engine logging solution

Mon, 24 Feb 2025 00:00:00 +0200

tl;dr: Capsa makes Unreal Engine logging a lot better, especially for those working on multiplayer. Check it out here .

I’m very excited to share what I’ve been working on over the last few months by announcing the v0.1 release of Capsa.

Introducing Capsa

An open-source Unreal Engine logging solution.

Ever since I started working on multiplayer games, I was quite surprised how few tools were available for working effectively with Unreal Engine logs for multiplayer games. After working on various titles, Mark Jawdoszak and I started building a tool called Capsa to address this. We’ve made this fully open-source so it can be used by other teams facing similar challenges.

More in-depth reasons why we’ve built Capsa and licensing can be found in this blog post on Capsa.gg .

About Capsa

Capsa makes this easy by automatically collecting logs and making them accessible in a structured, searchable, and shareable way.

Some of the features Capsa offers:

Logs with syntax highlighting and sharable links
Filter, search, and link logs
Merged client-server logs
Minimal setup, no game core or engine changes
Cloud-native, so host anywhere
Optimized for performance
Open-source, web stack AGPL3.0 and UE plugin MIT
Dark-mode support

Working on Capsa has been a blast and we’ve got tons of ideas of new features we want to add over the next few months.

To read the full announcement post, check out Announcing Capsa .

Building Capsa

Some notes on building Capsa and the tech we have used.

Extending Unreal Engine

We chose to extend Unreal Engine rather than replace functionality. We are hooking into GLog as an output device, so we can capture UE_LOG calls and we don’t require game code changes with a new log system. We want to have a system that is easy to add to and easy to remove from games, and I think this is a great way to achieve that.

The received log lines are formatted according to our Capsa log format so the server can parse it correctly. This way we don’t change any default Unreal Engine log output, but we receive the correct format on the server

Golang server log parsing

The log is processed in O(1) complexity, reading one character at a time. A buffer is built with all line characters until encountering a \n. The line buffer is then used to extract metadata for that specific line. The metadata is stored in the database and the log chunks are stored in S3-compatible storage.

Due to the per-character algorithm for log chunk processing, it allows the server to be very performant with extracting the metadata. At the time of writing this, parsing a 100k line log chunk (which is much bigger than will be done in production) takes just around 60ms.

More details on log processing can be found on Log processing .

Server-side merging and filtering

We had to make an important decision on doing log merging and filtering, whether we wanted to do this on the server or client. There are pros and cons for each, but we decided to do this server-side.

By doing this, we can re-use the existing log parsing code for incoming logs, get better performance, but it means that changing the filtering/merging settings requires reloading the full log from the server. We figured we needed server-side logic anyway, as large logs cannot be kept fully in the browser’s memory, so at least for v0.1, all of the merging and filtering logic is handled server-side.

Because we store per-chunk metadata, we can ignore chunks that don’t fit the filtering criteria as well, giving us a good performance boost. The benchmark for filtering lines from a 100.000-line chunk for streaming to a user (as is used for the admin panel) is around 70ms.

Golang server SQL

For handling database logic, I chose to go with sqlc and golang-migrate, as I have written in my Golang SQL handling article . This has been a good decision so far, as I have full control over the SQL to optimize for performance. I have already done a few optimizations in the queries, which would not have been possible with ORMs.

JWT with private/public keys

Authentication is hand-rolled in the API server, so there is no reliance on external parties for this. JWTs are used for this. Instead of using a signing secret, a private/public keypair is used.

This allows the web panel to check the JWTs (stored as a cookie) if they are valid and have users reauthenticate if they are not.

All API requests are done from the browser directly to the API server to not unnecessarily increase server load for the web panel. Yet we still want to validate the JWTs on page load to make sure we don’t perform API requests that will return in a 401 or 403 response.

Building Capsa cloud-native

Another concern when building Capsa was making it possible to deploy anywhere. This is why I have chosen to build the web stack with Golang and NextJS. Deploying these to any Linux or Windows system is very trivial. Both of the applications are available as Docker images and can be hosted anywhere by setting some environment variables.

Log processing web worker

The web panel uses a web worker to process incoming logs, to not block the main thread, which would make the website feel a lot slower due. This is achieved by writing a class to manage the web worker, which is then exposed in a React hook to work with the rest of the logic.

Runtime JSON validation

For the web panel, we have added runtime type validation by using Zod. The React code uses useSWR to call API endpoints as hooks. Instead of using API endpoints directly, we have added wrappers for requests, for example const { trigger, isMutating } = useAddTitle(); to add a new game title. This hook internally calls the API endpoint with useSWR, validates the JSON it gets back with Zod, and sets an error if it’s not valid.

By doing this, we can avoid runtime type exceptions.

Continuing development

We have tons of ideas on how to improve or extend the current functionality or new features we can add. But before we add this, we want to make sure we are on the right path with Capsa.

Now v0.1 is released, we will be integrating this into a few Companion Group projects and gather feedback from there, as well as early-adopters. We will continue to fix bugs and work on existing and new features over time, to slowly work towards a v1.0 release.

For v1.0 we have a few feature ideas that would make Capsa even more powerful to improve the lives of developers working on Unreal Engine multiplayer games. We are working on a few proof-of-concepts for this, though these will take some time as we need to figure out details, as we don’t want Capsa to take over game codebases, we want to keep Capsa non-intrusive.

Want to become an early adopter?

If you are interested in becoming an early adopter, the details on getting started are in Announcing Capsa . You can also reach out to me if you need any help, I’m more than happy to help you get started with Capsa.

All feedback - positive or negative - is very much welcome as well!

Why Baldur's Gate 3 is a masterpiece

Sun, 23 Feb 2025 00:00:00 +0200

This article does not contain spoilers.

I should preface this by saying that I have never played Dungeons and Dragons in my life, and knew very little about it before playing Baldur’s Gate 3. I started playing the game because it was recommended to me by my partner and I was somewhat skeptical about it, because it was so foreign to me. It was nothing like I expected, in the best way possible.

I am not a huge gamer. Even though I work on games, I tend to enjoy that more than actually playing games. I am quite picky with the games I play. There are quite a lot of games that I have enjoyed a ton, but nothing comes close to Baldur’s Gate 3, it is by far the best game I have ever played.

My first playthrough took me around 80 hours, where I took my time but didn’t do most side-quests (which I didn’t even know existed). My partner’s playthrough is, at the time of writing, 196 hours long and we’re only now moving into the end game ¹. I have also started a few playthroughs to explore different classes and approaches to the game at higher difficulties. It will be a long time before I’m done playing this game.

For brevity, I’ll abbreviate Baldur’s Gate 3 to BG3 going forward.

Endless choice and replayability

Having choices is of course a core part of RPGs, but BG3 takes this to almost an extreme. You don’t feel forced to take a particular path in your playthrough and there are no predefined paths you can take.

Choices you make early on can have a lot of influence on later parts of the game. The options you are given are not just to get a particular response from an NPC, but actually have a huge impact on the world in later acts.

No two playthroughs are the same. You can play as a murderous psychopath, a charming bard, or even as a cat. Even if you pick the infamous stealth archer min-max build, it’s highly likely two playthroughs will be radically different, even when you follow an Honor Mode guide to complete the game.

Romancing characters is done incredibly well and offers a ton of choices as well. You cannot romance all characters at once, and romancing a different companion offers even more value to start another playthrough.

On the topic of romance, I feel obliged to mention the ‘bear scene’ (if you know, you know). The fact that Larian took the time to add this as an option to me is incredible, regardless of which choice you made if you encountered this, the fact it’s there is already amazing to me.

Cohesiveness of the story

It’s incredible to me how cohesive the story is. With so many different ways to do missions, get somewhere or handle situations you would think there would be many inconsistencies later in the game.

Sure, there might be some tiny inconsistencies here and there, but to me it’s amazing how a game where the story is not linear apart from the three Acts can make the story of any playthrough feel like it was written to be played this way.

The overarching story of BG3 has some of the best storytelling that I have ever seen in a game, even better than most movies or TV series. Everything comes together, there are many plot twists that all make sense.

Everything in BG3 is interconnected. There are no parts of the story or world that feel out-of-place. Random NPCs you encounter in Act 1 will often reappear later in the game (unless you kill them for fun).

Having a game with such freedom of choice with a better, more cohesive story than most TV series and movies is nothing short of amazing.

Character development

The character development in BG3 is phenomenal, for companions as well as NPCs, both with phenomenal voice acting. Even side characters you meet when making certain choices in Act 1 will show up in later acts, and have gone through some development. The development of the dream visitor is amazing as well.

The villains in the game are so incredibly well-written. The villains are not just your everyday antagonists, they have complex personalities, there is character development for them during the game, tons of backstory and with some you can even work together if you like.

And that is such an amazing part of the game to me as well: there is such a degree of freedom to morality. Without getting too much into the political influence of modern games, I think BG3 does an amazing job of offering choices without forcing anything upon the player, which I think is the best approach, regardless of what your personal preferences are.

To expand a bit upon the complex personality of NPCs, one thing I dislike in some games is when all characters are super obviously ‘good’ or ‘bad’. This is definitely not the case in BG3. Yes, there are some ’evil’ characters, but even a literal devil in the game is presented in an almost ‘good’ way in some parts. This moral ambiguity adds another amazing layer to the story.

The companions deserve mentioning here too, they all have their own storyline with tons of choices, will (dis)approve of your choices and their personalities can change drastically based on decisions made. A part that I have not played myself, but only seen in playthrough videos is the fact that for many characters, you can play with them as the main character, unlocking more unique content.

Attention to detail

The attention to detail in BG3 is absolutely insane, which shows in many ways. A great example is that there are some voice lines that you will only hear in a modded playthrough, as your character enters a scene in a state that isn’t possible in the vanilla game. Even in the vanilla game there are tons of hidden details that most players will miss.

Larian added some details in the game that they knew less than one percent of players will ever see, some might still be undiscovered. This is such a stark contrast to some released games that feel unpolished, or some even unfinished, until a few months after release.

A game like BG3 obviously needs a lot of failsafes, think of cases where crucial NPCs get killed, locking you out of progressing a certain quest. The failsafes in BG3 are very subtle, they don’t limit your freedom to kill certain NPCs and even the failsafes have failsafes. I have had a few nights where I binge-watched videos about the failsafes in BG3 and it’s still incredible to me how many edge cases were covered.

The attention to detail is also clear with NPCs in the game. Nearly all NPCs have voice lines, most you can talk to, and quite a few completely random NPCs will offer some sort of side quest (that of course fit into the rest of the story really well). The banter between companions is another great touch as well.

Post-launch patches

To me, there is a very big difference between post-launch patches to make the game playable versus patches that improve the game. I won’t be naming any games, but there are some prominent examples of games that were released and needed a few patches (not counting the day-one patch) just to be playable. BG3 was not one of them.

Larian did release patches after launch, but these were mostly to improve the game further and take care of community feedback. Some amazing features have been added here, like Honor Mode and the epilogue scenes. In addition, some voice lines have been replaced, they were already fine, but the fact Larian took the time to improve further on this to me is great.

Mod support

Unofficial mod support was already available, but in a post-release patch, Larian added native mod support. Right now, it’s even possible to have cross-platform multiplayer sessions with mods enabled (of course with some platform restrictions). The mods add a ton of new content and possibilities, adding even more replayability to the game.

Music

I’m a huge fan of the soundtrack of BG3. The music fits the game incredibly well. The song of the boss fight with one of my favourite characters of the game, Raphael, still gives me goosebumps when I hear it. Then there’s also the fact that this character stars in his own boss song, which is amazing to me as well.

Lack of microtransactions

One of the things BG3 does that I hope will be adopted by other games is the lack of microtransactions. There are no DLCs and no special currencies purchased with real money. You buy the game and that’s it. All post-release content was made available to all players.

The lack of microtransactions is another way to see that BG3 was made to be an amazing game, not as a corporate cash-grab. I’m not saying BG3 could never be made with investors being involved, but the freedom Larian had regarding monetization by not having to report to investors cannot be ignored.

Conclusion

Baldur’s Gate 3 has set a new gold standard for RPGs. In my opinion, this is, so far, the closest we have ever gotten to a ‘perfect’ RPG. Whether you tend to play casual or are into min-maxing your builds at the highest difficulty, BG3 is a game worth playing, over and over.

BG3 sets an example that I hope many games - both RPGs and other genres - will follow. There are many conclusions you can draw from BG3’s success about the games industry as a whole.

To me, it will be a game that I will keep playing for a long time. With no DnD experience there is a ton for me to discover in the base game already, and after that there will be tons more with the official and unofficial mods.

After 214.5h we finished the game on March 23, 2025. ↩︎

Finding the best translation for Marcus Aurelius' Meditations

Sat, 15 Feb 2025 00:00:00 +0200

I have read quite a lot of book on Greek and Roman philosophy and Marcus Aurelius’ Meditations is the book I have read most often by far. In total, I own around 7 translations and I have read sections of other translations as well. In addition to this, I have read several secondary works, in addition to the sections of the Meditations that I have read in the original Greek. In 2021/2022, I have hosted an eight-month long weekly reading group on the Meditations as well.

One important note about reading philosophical works, is that translation is a very important aspect if it was written in a different language. There are many features present in Ancient Greek (the language that Meditations was written in) that are not present in English, or words with no English equivalent.

Translation is not just about translating the words but - especially with philosophy - it is also important to capture the style, tone, etc. of a text as this is essential to communicate the meaning of the source material. If that is not possible due to linguistic challenges, that should be clarified by the translator.

So, back to Marcus Aurelius. After reading the Meditation and several secondary sources, what is my favourite translation of the work in English?

Waterfield’s Annotated Edition

By far the best translation of the Meditations in English is Robin Waterfield’s Annotated Edition.

In this translation, Waterfield does an amazing job of capturing the conciseness of the original work. The English text ‘feels’ very similar to the original, literary, yet direct.

In addition to the great translation, this edition has a great introduction, clarification on translating certain terminology of Greek philosophy, but to me the best part are the tons of annotations present as footnotes.

The Meditations was not a book to be published, for Marcus Aurelius, it was most likely his “commonplace book” and Waterfield does an amazing job of adding connections to other works works and even to similar passages within the book, or simply explaining cultural references.

If you are interested in reading Meditations, or if you have read it before but want to understand it better, you cannot go wrong with this edition.

The ISBNs are:

Hardcover: 9781541673854
Paperback: 9781541673861

Other translations worth checking out

If you can read Ancient Greek: Haine’s translation from the Loeb Classical Library is not an amazing translation, but has the original text on the left, and the English translation side-by-side on the right, which can be great.
Dutch translation: I have not read as many Dutch translations as English ones, but I like Simone Mooij-Valk’s “Persoonlijke notities” best so far.
German Translation: there is a German translation of Waterfield’s edition (Selbstbetrachtungen: Die kommentierte Edition, translated by Elisabeth Liebl), which I have partly read and quite liked.

ἐξετάζειν τί μοί ἐστι νῦν ἐν τούτῳ τῷ μορίῳ, ὃ δὴ ἡγεμονικὸν καλοῦσι, καὶ τίνος ἄρα νῦν ἔχω ψυχήν;

At this moment, what is occupying that part of me they call the command center? What kind of a soul do I actually have at the moment? (Waterfield’s translation of Meditations 5.11)

My favourite way to handle SQL in Golang

Fri, 14 Feb 2025 00:00:00 +0200

tl;dr: using sqlc and golang-migrate allows plain easy SQL in your application

Golang is a great language when building web/API services. And that makes a lot of sense, as Go was created for building scalable, performant applications. The language is very cohesive, compiles fast, has great performance and has a great ecosystem in addition to its rich standard library and great toolchain. Whenever I need to build a web service or API, I tend to reach for Go when possible, hence why I built quite a lot of applications with it so far, including around a dozen or so application back-ends.

But this is not an article about why Go is great, but rather on how to handle SQL. Considering Golang is not made to be used in Ruby on Rails-type frameworks, you’ll most likely have to set up the SQL-logic yourself.

My approach to SQL handling in Golang is the result of trying out a lot of approaches and seeing what works and what doesn’t. This approach is mostly aimed at building API services, backed by a database. In my case, I use Postgresql, though other SQL databases should also work with a few tweaks.

As most applications that I build deal with quite a lot of data, I don’t want to work with ORMs like Gorm, due to the performance impact and unnecessary abstractions that it introduces. In general, I’m not the biggest fan of ORMs, as I already know SQL and don’t like learning additional tools to abstract away the actual SQL code, using SQL directly for me is simply faster, both in terms of performance as well as writing the application.

So how then do I approach SQL in Golang?

Code generation with sqlc

One of the things I like about the Go ecosystem is that there are a lot of code generation tools. Code generation is something that is quite underused in my opinion, and sqlc is an amazing example of how code generation can be very powerful.

With sqlc, you can write plain sql, and then run a command that generates the Go code in a module you can use in your application.

Configuring sqlc

After installing sqlc on your device, getting up and running with sqlc is quite trivial. The generic installation instructions are found in the sqlc docs , but I’m using a bit of a modified configuration.

We start with creating the sql/ and migrations/ folders, for SQL queries and database migrations.

In the migrations directory, add a file 0000_schema.up.sql and 0000_schema.down.sql for adding your initial database schema, take this up migration for example:

BEGIN;

-- Enable uuid extension
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

-- Users table
CREATE TABLE users (
 -- User fields
);

-- The rest of your schema

COMMIT;

Make sure to add the down migrations as well. The migrations are used by sqlc to generate the database structure and add typed columns for database manipulations. They will also be used to perform actual database migrations. Wrap your migration logic in BEGIN and COMMIT so you can ROLLBACK on a failed migration.

Full sqlc config example

Here is the full configuration I’m using for the Capsa API service:

version: "2"
sql:
 - engine: "postgresql"
 queries: "sql/"
 schema: "migrations/"
 gen:
 go:
 package: "database"
 out: "internal/data/database"
 sql_package: "pgx/v5"
 emit_json_tags: true
 json_tags_case_style: "camel"
 emit_pointers_for_null_types: true
 overrides:
 # UUID
 - db_type: "uuid"
 go_type:
 import: "github.com/google/uuid"
 type: "UUID"
 - db_type: "uuid"
 nullable: true
 go_type:
 import: "github.com/google/uuid"
 type: "UUID"
 pointer: true
 # Timestamp
 - db_type: "pg_catalog.timestamp"
 nullable: true
 go_type:
 import: "time"
 type: "Time"
 pointer: true
 - db_type: "pg_catalog.timestamp"
 go_type: "time.Time"
 # Custom
 - column: "logs_chunks.category_counts"
 go_type:
 import: "github.com/capsa-gg/capsa/server/internal/entities"
 type: "LogChunkMetadata"
 - column: "logs_chunks.severity_counts"
 go_type:
 import: "github.com/capsa-gg/capsa/server/internal/entities"
 type: "LogChunkMetadata"

This configuration replaces the pgx (Postgres database driver) UUID and Timestamps with the Google and standard library implementations respectively. It also specifies a few column types to be used, so the returned data has better compatibility with my domain logic. For further explanation on this, see the docs .

Writing SQL statements

With sqlc configured, we can write some SQL code! The docs explain things in more detail, but here are two very simple examples:

-- name: GetUserByID :one
SELECT * FROM users
WHERE id = $1;

Gets a user by their ID. The * return value here will use the database schema to determine the return type to make it fully typesafe.

-- name: UpdateUser :one
-- Update a user with optional parameters
UPDATE users
SET first_name = @first_name,
 last_name = @last_name,
 user_role = @user_role
WHERE id = $1
RETURNING *;

For updating a user, this also shows how arguments can be named with sqlc for clarity in the domain logic.

Complex SQL

The two examples above are the ‘hello world’ equivalent of SQL statements. In larger applications things become more complex. An example of this is searching. Although there might be better ways, this is the way I have implemented search for Capsa, which is not too complex:

-- name: SearchResources :many
-- Searches in some database tables to find matching resources based on a "contains string" pattern (LIKE '%<arg>%')
WITH resources AS (
 SELECT
 'Environment' AS table_name,
 lower(e.key::text) AS identifier,
 concat(e.name, ' environment for ', t.name) AS description,
 '' AS details
 FROM environments e
 LEFT JOIN titles t ON t.id = e.title

 UNION ALL

 SELECT
 'Logs' ,
 lower(l.log_uuid::text) ,
 concat(l.log_type, ' log on ', platform, ' (', t.name, ', ', e.name, ')'),
 count(lc)::text
 FROM logs l
 LEFT JOIN environments e ON e.id = l.environment
 LEFT JOIN titles t on t.id = e.title
 LEFT JOIN logs_chunks lc ON l.id = lc.log
 GROUP BY l.log_uuid, l.log_type, platform, t.name, e.name
)
SELECT table_name, identifier, description::text, details::text
FROM resources
WHERE identifier LIKE '%' || lower(@search) || '%'
LIMIT sqlc.arg('limit');

Another more complex example is filtering with optional arguments. The full statement is over 50 lines long, but here is the most important part of the implementation for Capsa:

-- name: ListAvailableLogs :many
-- Fetches all log chunks and aggregates an overview.
-- LogUUID is an optional field used as a filter, which if set will return only a single result.
WITH -- Omitted
SELECT -- Omitted
FROM logs l
JOIN cat_counts cc ON cc.log = l.id
JOIN sev_counts sc ON sc.log = l.id
JOIN chunk_data cd ON cd.log = l.id
JOIN environments e on l.environment = e.id
JOIN titles t on e.title = t.id
LEFT JOIN links ll on l.id = ll.source
WHERE ( l.log_uuid = sqlc.narg(filter_by_log_uuid) OR sqlc.narg(filter_by_log_uuid) IS NULL ) -- Optionally filter by Log UUID
AND ( e.key = sqlc.narg(filter_by_environment)::uuid OR sqlc.narg(filter_by_environment) IS NULL ) -- Optionally filter by Environment
AND ( l.platform = sqlc.narg(filter_by_platform)::varchar OR sqlc.narg(filter_by_platform) IS NULL ) -- Optionally filter by Platform
AND ( l.log_type = sqlc.narg(filter_by_logtype) OR sqlc.narg(filter_by_logtype) IS NULL ) -- Optionally filter by LogType
GROUP BY l.id, t.name, e.name, cd.line_count, cd.chunk_count, cd.earliest_start, cd.latest_end, ll.sum
ORDER BY earliest DESC
LIMIT @fetchlimit::int;

This will simply ignore all arguments that are not set, or include them if they are. In all honesty: the syntax with an ORM would be cleaner, but I do suspect the performance will take a hit. If performance with ListAvailableLogs becomes an issue, I can optimize it, with ORMs, that is a lot harder.

Generating the code

To generate the code, simply run sqlc generate in the root of your project, and you should have your code! There are more commands you can use for analysis and linting, which are outlined in the docs .

Using the generated code

During the application start, you can use your config to generate the instance of the sql-generated structs:

package example

import (
 "github.com/jackc/pgx/v5/pgxpool"
 "github.com/capsa-gg/capsa/server/internal/data/database"
)

// NewDatabase initializes a Database instance.
func NewDatabase(c *entities.Config) (*database.Queries, error) {
 ctx := context.Background()

 // Database connection
 dbConn, err := pgxpool.New(ctx, c.DatabaseConnectionString())
 if err != nil {
 return nil, fmt.Errorf("error opening database connection: %w", err)
 }

 // Ping database
 err = dbConn.Ping(ctx)
 if err != nil {
 return nil, fmt.Errorf("error pinging database: %w", err)
 }

 // Database instance
 db := database.New(dbConn)

 return db, nil
}

You now have a *database.Queries instance that you can use to use query the database! Using it is as simple as

user, err := db.GetUserByID(ctx, userId)

There is also support for performing logic inside of a transaction, as outlined in the docs .

Handling migrations

You should only be changing the database structure using migrations. We already have the migration scripts used by sqlc, which we can use to perform the database migrations.

For migrations, I’m using the golang-migrate package, with a small wrapper. This is the full code of the migrator package for Capsa:

package migrator

import (
 "database/sql"
 "errors"
 "fmt"
 "net/http"

 "github.com/golang-migrate/migrate/v4"
 "github.com/golang-migrate/migrate/v4/database/pgx/v5" //nolint:gocritic,stylecheck // Needs import for using pgx.WithInstance
 "github.com/golang-migrate/migrate/v4/source/httpfs"

 // Needs side effect from pgx/v5.
 _ "github.com/golang-migrate/migrate/v4/database/pgx/v5" //nolint:gocritic,stylecheck // Needs import for side effect

 "github.com/capsa-gg/capsa/server/migrations"
)

// Direction indicates the migration direction.
type Direction string

// UpAll and DownAll indicate the migration direction.
const (
 UpAll = Direction("UpAll")
 DownAll = Direction("DownAll")
)

// New returns a migrator closure that will accept UpAll or DownAll to up or down migrate the database.
func New(dbConn *sql.DB, dbName string) func(Direction) error {
 return func(direction Direction) error {
 // Direction check
 if direction != UpAll && direction != DownAll {
 return errors.New("migration direction should be 'UpAll' or 'DownAll'")
 }

 driver, err := pgx.WithInstance(dbConn, &pgx.Config{})
 if err != nil {
 return fmt.Errorf("invalid target postgres instance, %w", err)
 }
 // Source instance for the migrations embedded in server binary
 sourceInstance, err := httpfs.New(http.FS(migrations.Migrations), ".")
 if err != nil {
 return fmt.Errorf("invalid source instance, %w", err)
 }

 // Create migrator instance
 migrator, err := migrate.NewWithInstance("httpfs", sourceInstance, dbName, driver)
 if err != nil {
 return fmt.Errorf("failed to initialize migrate instance, %w", err)
 }

 // Do the actual migration
 if direction == UpAll {
 return handleMigratorErrors(migrator.Up())
 }

 if direction == DownAll {
 return handleMigratorErrors(migrator.Down())
 }

 return errors.New("you should not see this")
 }
}

func handleMigratorErrors(err error) error {
 if errors.Is(err, migrate.ErrNoChange) { // Do not report error when no database change
 return nil
 }

 return err
}

With this, performing the database migrations is as simple as

import "path/to/migrator"

migrate := migrator.New(db, config.DatabaseName)
err = migrate(migrationDirection)

Now you have good ‘ol plain SQL for migrations as well as database queries in Golang with code generation!

Example to see this in action

An example of this approach in action can be found in the Capsa API server . Capsa is still work in progress, but I highly doubt the SQL approach will change.

Easy help command for Makefiles

Thu, 13 Feb 2025 00:00:00 +0200

I love Makefiles, they are my prefered way to manage builds. Not just for C or C++, I also use it for Golang for example. To me, it is the best way to manage build commands in a single place.

As I love CLI tools in general, and working with them, the --help flag is often a great help, but Make does not offer this out of the box, and navigating complex Makefiles can be challenging.

I can’t remember where or when, but a few years ago, I found a snippet somewhere of a Makefile command that adds a coloured text output of all Makefile commands. Using it is as simple as making the following your first Makefile command:

.PHONY: help
help: ## Shows all commands
 @echo 'All Makefile commands:'
 @grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

After adding this, you can simply add comments after your other Makefile commands with two # characters, like this:

.PHONY: dev
dev: CFLAGS = $(CFLAGS_DEV) ## Development build for /src and /example
dev: clean $(LIB) $(EXAMPLE)

This will now be automatically generate help output when running make help, or when you have defined this as your first Make command, with simply make

Hello, world! Again!

Wed, 12 Feb 2025 00:00:00 +0200

Recently, I have had quite a lot of ideas about interesting blog posts, some short, some long; some general notes and some more technical. All in the hope that this can be helpful or at least somewhat insightful for others.

I have updated my website to add slightly better blog support, including categories and tags to make this possible, plus better RSS feed support.

Currently, the website is quite basic. And although I have considered giving it a bit of an overhaul with better styling, I would prefer to keep this website somewhat minimalist. I might make some changes to improve readability, but the styling itself will still be very minimalistic.

Anyway, that’s it for now!

Webinar replay: common pitfalls for software start-ups and how to avoid them

Thu, 13 Jan 2022 00:00:00 +0200

On November 29th, 2021, I gave a webinar for Bytecode about common pitfalls for software start-ups and how to avoid them.

A replay of this webinar can be seen here:

Runtime JSON typechecks with Typescript interfaces

Mon, 24 Aug 2020 00:00:00 +0300

The contents of this article are technical in nature and mainly written for software developers

Bytecode is currently working on a project for a startup. We are developing an app that is linked to a backend data storage service. For this project, I wanted to have more security around the API calls and validate the types at runtime.

Background

At Bytecode, we use Typescript extensively for front-end and mobile development, to prevent Javascript’s liberal dynamic typing system from causing errors. Typescript adds an extra layer of security. In recent years, we have seen a strong decrease in type-related errors due to the use of Typescript.

However, Typescript also has it’s limitations, the main one being that types and interfaces disappear during compile-time. So there are no run-time checks on external data that is not available during compilation, like for example API responses. Ideally, Typescript would support marshalling in a way similar to Go , to secure type safety during runtime. Unfortunately, however, the Typescript layer disappears during compile-time and only Javascript remains: code that knows nothing about the types and interfaces defined in the source code. Unfortunately, the Go-like approach would never be possible using features built into the language.

Until recently, we did our API calls directly within Redux actions, which didn’t cause any problems for small applications. However, this setup is not the best, considering the single-responsibility principle and keeping in mind that Redux actions can get confusing with large projects, if you’re not careful. This is why we recently switched to creating separate API packages as an abstraction layer on top of the API calls. We now only call a function that executes the API calls and checks HTTP errors. If there are no errors, we’ll get the data back. If there are errors, an error is thrown. This way, the Redux code does not know anything about the details of the API call.

Another reason to use a separate API package, is our desire to be able to set up an SDK more easily later on, when we start targeting another platform (think of an application, first only web, but where a mobile app will be added as well). It will then be possible to use shared code easily, without duplicate logic. Since these API packages become increasingly important with more dependent applications, the run-time guarantees also get more crucial. If the SDK says that a function returns a certain data type, we also want to guarantee this or else give an error message.

Requirements and research

Our research question consisted of several parts:

How can we generically check a JSON object against a Typescript interface, without the need for duplicate code for type definitions?
How can we achieve the above without having to modify other production code outside the API package to make this check possible?
How can this be done in NodeJS, React Native and in the browser?

There are enough libraries that make it possible to check a JSON structure, based on a DSL (domain specific language). However, this wasn’t what we were looking for, because we were already using Typescript and didn’t want to maintain the same type definition in multiple ways. We would prefer to develop a solution where no code generation or extra step in compilation is needed, but everything on-the-fly during runtime (like Go).

A few months ago, I read a blog post by Picnic , describing their project “Aegis”, in which they had offered a solution to this problem. However, I noticed that it was still difficult to implement. The code is open source, but there was no example of implementation on a larger scale, because this was done within Picnic’s proprietary app. A code generation step was also needed. We would, if possible, prefer not to have this extra step.

On the Subreddit of Typescript I had placed a post , where I submitted my question. I primarily received responses with examples of code generation solutions. Some responses discussed runtime solutions, but these solutions were unnecessarily complex and/or required modifications within the build configuration of Typescript. We prefer to avoid this, because we prefer to keep something experimental separate from the rest of our production code, so that if we are not satisfied, we can revert the changes.

A possible solution that popped into my head was the following:

Load all type-definitions through the file system as strings
Use the Typescript compiler as production dependency and parse these strings
Compare the result of parsing against the JSON data to see if it matches the interfaces

However, this would mean that a substantial part of the Typescript compiler would have to become part of the app and thus increase the bundle size. The Typescript compiler is not the fastest in the world either, so this would take a considerable amount of extra time when it has to be done on-the-fly. In addition, the filesystem is only suitable for Node.js and not for browser environments, so compatibility could not be maintained. Unfortunately this solution was not feasible.

Ultimately, I chose to use Picnic’s Aegis , mainly because of the simplicity of the tool and because it can be used without modifying other aspects of the project (compilation steps, configurations or production code in other parts of the application).

Implementation

The final implementation is as follows. Within the API folder, all public types (that is, the arguments and return types of the entire API package) are defined in the types folder. For all types in this folder, Aegis creates decoders and stores them in the internal folder of the API package.

In order to make the aforementioned solution usable for Bytecode, a few adjustments had to be made in Aegis. For example, we added ESLint comments at the top of the file. This was done in a fork of Aegis on Github. This is the dependency used in Bytecode’s project. To build the decoders, a command has been added to the package.json of the React-Native/Expo project. By simply running yarn run aegis, Aegis is called with the right arguments and all decoders are built.

The production code of the API package already used an internal returnOrThrow function, which received an internal API response type (consisting of the response of the API and/or an error if it occurred), threw an error if it existed and otherwise returned the data. This function has been modified so that a second argument is given to the function, namely the decoder. In returnOrThrow the decoder is then used to check the data before it is returned. See the example below:

import { Decoder } from "decoders/types";
import { guard } from "decoders";

interface APIResultSuccess<T> {
 data: T;
 error?: undefined;
}
interface APIResultFailure {
 data?: undefined;
 error: string;
}
type APIResult<T> = APIResultSuccess<T> | APIResultFailure;

const throwOrReturn = <T>(result: APIResult<T>, decoder: Decoder<T>): T => {
 if (result.error) {
 throw new Error(result.error);
 }
 // We can assume that data is valid (type T) if no error was found
 const data = result.data as T;

 const decodeChecker = guard(decoder);
 const _ = decodeChecker(data); // Throws if it's not valid
 return data;
};

export default throwOrReturn;

If the return body does not satisfy the decoder, an error is thrown, which can be caught when calling the API call.

Wishlist

For now, the implementation of JSON type checks is still experimental. Within the codebase, this type check only affects a small (separate) part, so it can be easily removed later on. This is why it is not automated yet, something we would like to add later on. Automating this without modifying the build configuration can still be a challenge. Until we automate this completely, we can add a check in the CI pipeline that gives an error message when running Aegis causes file changes in Git, meaning the decoders have not been updated after making changes to the type definitions.

Another very good use-case of this setup would be the end-to-end testing of the API for which the package is built. The end-to-end testing of APIs is something that has been on the Bytecode wishlist for some time now. JSON type checking can also provide great added value, allowing you to immediately check whether the API returns data according to the expectations.

Regarding improvement to Aegis itself, at the moment Aegis is mainly built for the “happy flow”, there are still some edge cases that don’t quite work. The tool is now definitely usable, but improvements are still needed for large-scale use.

Support for Aegis configuration files is another useful addition. Now arguments for importPath and outputFile must be given as CLI options. Simply calling aegis generate, which then loads the configuration by itself would be a nice addition.

If it turns out that this workflow works very well for Bytecode, there is a good chance that we will further develop the Aegis tool ourselves, open-source of course. For now, the tool is not yet available on NPM. That would be the first step towards a stable release.

Example project

As an addition to this article, an example project is also available, with a simple API call, where the tool can be seen in action.

Check out the project on Github .

How the corona pandemic might permanently change our work-attitude

Tue, 28 Jul 2020 00:00:00 +0300

This article is a translated and slightly modified version of the article I originally wrote in Dutch for Bytecode. To read the original, click here .

Whether we like it or not, the corona pandemic is far from over. Even though quite some countries are slowly going back to “normal” (albeit with some more distance between each other), others have yet to reach the peak of their first wave.

We should not forget that, for most people, it will take a while to go back to the large office buildings, nearly hugging each other in the elevator while going to the meeting with everyone in person, cramming 8 people into a tiny room. Should we actually go back to this?

Working before corona times

If you think about it, the current mindset about work is pretty crazy. Why do I have to be in the office to do work that I might be able to do a lot more efficiently in other places? Why do I still have to go to the lecture hall, where I can’t skip parts I already know and where I can’t rewind if I want to hear something again (without delaying the lecture for others)?

Before the corona crisis, few people were lucky enough to have the opportunity to decide their own approach to work. Mostly, remote work is only allowed when ill. Starting earlier in the morning, so that you can finish earlier in the day? No chance of success. Starting later and finishing later because you’re not a morning person? Certainly not.

Our mindset about working can be improved

Is it really necessary to work in this “pre-corona” way? Do you have to be in the office for everything? Of course not everyone can (or wants to) be a digital nomad , flying across the world to work online in hotels. However, there are possibilities to make work more enjoyable for everyone. I don’t envision a shelf filler, car mechanic or electrician who works from home at times that suit him. But a translator, consultant or marketeer?

Personally, I am very attracted to the ideas from the async manifesto , a set of insights on how software development can be better implemented. The ideas come down to this: use modern tools, create a flexible work environment, do not disturb people’s concentration unless it’s urgent and only hold meetings when it’s really necessary. The async manifesto is focused primarily on software developers, but I think anyone with a non-physical profession can benefit from the tips given here.

When it comes to productivity, it seems that working from home (in a quiet office) is a lot more productive than working at the office. It is fairly easy to understand why this is the case for many people. Every time you are interrupted in a “deep-work ” session, it takes 25 minutes to get your focus back. Not to mention the “this-could-have-been-an-email”-meetings that take an hour each.

How great would it be, instead of being constantly interrupted for some trivial question, to just answer non-urgent questions a few times per day? To not be disturbed by people who talk too loudly about yesterday’s soccer game? As Jason Fried explains in his TED talk , you have to be able to work long uninterrupted stretches to really get something done.

The corona pandemic offers an opportunity to reflect on the current way of thinking

Now that a large part of us is still working from home due to the corona crisis, we have a chance to see what really works for us. Is that 9-to-5 job really ideal, or does a 10-to-6 or 7-to-3 job fit better, perhaps? Or why not 7-to-4 with an hour’s break to run a lap? And if 9-to-5 office-based work turns out to suit you best, that’s fine too, of course!

How am I actually most productive? And more importantly, how do I actually get the most satisfaction from my work? These are questions to which we will be able to find an answer in the near future, because we now have room for experimentation.

Hopefully the changes in (higher) education will also have a lasting effect, because of the long-awaited technological innovation.

I’m not in favour of closing down all the offices and changing to remote work as a standard, which is naturally unfeasible. For example, I also think that face-to-face meetings work better than talking on the phone. But I believe that - with companies large and small - a lot can be improved compared to the current situation, if the right balance can be found. Let’s hope that this will be a positive lasting effect of this corona crisis.

Hello, world!

Sat, 18 Jul 2020 00:00:00 +0300

Welcome

Big thanks for checking out my personal blog!

As of today, my website is completely in English and I’ve also added my learning page on here, where I tell about my learning path to try and become a self-taught computer scientist.

With this, I’ve also added this blog on my website, where I’ll be posting from time to time about my experiences in learning, or to share information I hope will be valuable to others.

I’m also thinking about giving my site an overhaul, to improve the styling, readability and responsiveness and to add some features, but until this website is booming with traffic, I think this site will be more than just fine.

Anyway, that’s it for now!

Blogs by Luciano Nooijen

Creating and validating JWTs with public/private keys (JSON Web Key Set, JWKS)

JSON Web Key

Public/private key generation

Accessing the public and private key

Signing JWTs

Validating JWTs using the private key

Sharing the public key

Inspecting the JWT

Validating the JWT in NextJS

Seeing this in action

Why I stopped using AI code editors

Tesla FSD

My experience with AI code editors

The loss of Fingerspitzengefühl

You can get by without these tools

Where I draw the line

Doing what you love

My advice to new programmers

Conclusion

Announcing Capsa, an Unreal Engine logging solution

Introducing Capsa

About Capsa

Building Capsa

Extending Unreal Engine

Golang server log parsing

Server-side merging and filtering

Golang server SQL

JWT with private/public keys

Building Capsa cloud-native

Log processing web worker

Runtime JSON validation

Continuing development

Want to become an early adopter?

Why Baldur's Gate 3 is a masterpiece

Endless choice and replayability

Cohesiveness of the story

Character development

Attention to detail

Post-launch patches

Mod support

Music

Lack of microtransactions

Conclusion

Finding the best translation for Marcus Aurelius' Meditations

Waterfield’s Annotated Edition

Other translations worth checking out

My favourite way to handle SQL in Golang

Code generation with sqlc

Configuring sqlc

Full sqlc config example

Writing SQL statements

Complex SQL

Generating the code

Using the generated code

Handling migrations

Example to see this in action

Easy help command for Makefiles

Hello, world! Again!

Webinar replay: common pitfalls for software start-ups and how to avoid them

Runtime JSON typechecks with Typescript interfaces

Background

Requirements and research

Implementation

Wishlist

Example project

How the corona pandemic might permanently change our work-attitude

Working before corona times

Our mindset about working can be improved

The corona pandemic offers an opportunity to reflect on the current way of thinking

Hello, world!

Welcome