Flexday's clients ran on AWS, on Azure, and in their own data centres. They all needed the same permission model, and no single cloud provider could give it to them.
Flexday shipped several products to enterprise clients. Each product had its own UI and its own backend services, and every one of them needed to know what a given user was allowed to do. We built one service to answer that question, and everything else asked it.
The obvious move is to use whatever the cloud already gives you. It did not work here, for two reasons.
Our clients were not in one place. Some ran on AWS, some on Azure, some on their own hardware. Picking AWS IAM would have meant writing a second implementation for everyone else, so two sets of rules, two sets of bugs and two audit trails to explain during a security review.
Cloud IAM also governs cloud resources. It is very good at deciding whether a principal can read an S3 bucket. It has no idea what a purchase order or a shift roster is, and those are the things clients actually wanted to control. Entra app roles get closer but stay coarse, and per client custom roles get awkward quickly.
A tenant is one client organisation, and everything belongs to one. A user has an email and credentials, and can be a member of more than one tenant. The membership holds the role assignments rather than the user, so the same person can be an admin for one client and a viewer for another.
A role is a bundle of permissions, either a system role we shipped
or a custom one a tenant defined. Permissions are plain strings in the form
resource:action, so project:read or
invoice:approve.
Two tiers, because the two kinds of question cost different amounts to answer.
Most checks are coarse. Can this user open the billing page. That answer is already in the access token, so the service validates the signature against a cached JWKS and reads the permission out. No network call, and no dependency on us being up.
Some checks cannot be answered from a token. Can this user approve document 4192, which belongs to a project they may or may not be on. Putting every resource a user can touch into a token would make it enormous and stale within minutes, so those go to a central endpoint with the subject, action and resource, and we answer.
Splitting it this way decided something I had not expected. A decision made locally from a JWT leaves no trace with us, so anything that needs an audit trail has to take the central path whether or not it needs it for correctness.
Stateless tokens are fast and hard to take back. An admin removes someone's role and that person keeps the old permissions until their token expires.
Short expiry is the usual answer and it is a blunt one. Fifteen minutes of stale access is still fifteen minutes, and shortening it further just multiplies refresh traffic.
We used a policy version per tenant instead. Every token carries the version it was minted under. When an admin changes a role the tenant version increments, and services pick the new value up from a small cached map. If a token is behind, the service forces a refresh or falls back to a central check, so a permission change lands on the next request rather than at the next expiry.
Refresh tokens rotate on every use, and a reused one invalidates the whole chain. That is how a stolen refresh token gets noticed instead of quietly working for a month.
Other Flexday services and client integrations authenticate as themselves rather than as a user. Client credentials go in, a token with scopes comes back, on a separate audience so a service token cannot be replayed against a user endpoint.
Scopes are the same permission strings that roles are built from. One vocabulary across people and machines meant one place to look when something came back denied, which mattered more in support than it did in design.
The worst class of bug here is the silent one. A query missing its tenant filter returns another client's data and nothing fails. No exception, no alert, and the tests pass because they only ever seed one tenant.
Two things helped. Every repository function takes the tenant id as a required first argument and throws without it, so a missing filter becomes a crash instead of a leak. And the test suite seeds two tenants for every case, then asserts that a request scoped to one never returns a document belonging to the other.
The second one caught more than the first did.
Around 20 client tenants and 35k users, with roughly 8k active on a given day. Access tokens last fifteen minutes, so a working day is about 32 refreshes per user. That, plus the resource level checks, lands near 50 requests a second at peak.
A membership stores its resolved permission array rather than joining roles at read time, and the array is recomputed whenever a role changes. Minting a token becomes a single indexed read instead of a lookup chain, at the cost of a write fan out when someone edits a role that many people hold. Role edits are rare and token mints are constant, so that trade went the right way.