Chris Bao's Blog

Background

For modern applications, authentication usually follows a pattern where a dedicated, centralized service handles authentication and authorization—that service is the identity provider (IdP)—instead of every app implementing those concerns on its own.

Entra ID is the IdP for the Azure cloud platform. As I mentioned in my previous articles, you can use it to add authentication and authorization to your Azure solutions. Feel free to revisit those posts if you need a refresher.

In the AI era, people often say agents may work alongside—or even replace—humans in the workplace. Your next “colleague” at the company might be an agent. To support that trend, Entra ID extends its existing architecture with Entra Agent ID, acting as the IdP for AI agents to manage their identities, authorization, and more. In this article, I will walk through what I learned about the following questions:

• Why is managing Agent Identity so important?
• How does Entra Agent ID differ from classic Entra ID app identity?
• How do you create Entra Agent ID objects and use them for authentication and authorization?

If these topics interest you, bookmark this post and let’s get started. After reading it, you should understand the blueprint-to-agent token flow and where each object fits in Entra.

Before cloud and mobile apps, the network was the security boundary—the corporate intranet was treated as trusted. Today, you cannot reliably know who or what is on your network. What can you use as a boundary instead? The answer is identity as the security perimeter—at least the primary one. That idea existed long before AI agents showed up.

An identity is the set of attributes that define or characterize a person, a device, or a software component. Common categories include:

• Human identities: ordinary user accounts.
• Machine identities: further split into device identities (hardware) and workload identities (apps or services).
• Agent identities: AI-driven agents—the focus of today’s discussion.

When any identity—user, device, or application—requests access to a resource, that identity becomes the center of verification. The security question is no longer “Is this request coming from inside our corporate network?” but “Who or what is making this request, and should we trust it?” That is why identity is the security perimeter: every access decision starts with identity—verify it, then control what that identity is allowed to do.

From Entra ID to Entra Agent ID

An AI agent is still an application at its core, but autonomy and decision-making change the security model compared to traditional apps. That is why Agent Identity is modeled and managed as its own category.

Next, let’s compare traditional applications and AI agents from Entra’s architecture and design perspective.

Entra ID’s two-layer model

Entra ID manages application identity with a two-layer model: app registration and service principal:

• App registration: You define the app name and settings—scopes, app roles, redirect URLs after sign-in, and so on. If you know OAuth 2.0, this step is essentially registering the OAuth client. Entra ID supports OAuth 2.0 as the authorization server.
• Service principal: When a tenant adds the app, Entra creates an identity in that tenant that can take action. In Entra ID, this is the unit to which roles and permissions are assigned.

The app registration is not the runtime object. Registering an app creates a service principal in the tenant; roles and permissions are granted to the service principal. Think of the app registration as a template defined in a tenant; instantiation produces the service principal. Smart design, right? It is very much like the relationship between a class and an object in object-oriented programming.

An app registration can be published and managed in one Entra ID tenant while the app is consumed by many tenants—multi-tenant apps.

One more important point: in each tenant, the service principal can hold credentials (for example, a client secret) and use them to authenticate and obtain tokens.

Entra Agent ID’s three-layer model

Building on that design, Entra Agent ID introduces a three-layer model:

• Agent Identity Blueprint: For classic apps, the template/instance relationship was implicit. For agents, the name Blueprint makes it explicit. Conceptually it matches app registration: name and baseline settings, plus credential configuration used to validate agent-related identity flows.
• Agent Identity Blueprint Principal: Also a service principal—one blueprint principal per tenant. Up to this layer, the model closely mirrors the two-layer app model. The blueprint principal is created automatically; you do not provision it manually.
• Agent Identity: This is the new piece. An agent identity is also a service principal, created by the blueprint principal. It represents a single running instance of an agent—the work identity for that agentic application instance.

This architecture keeps one blueprint principal per tenant (the service principal for the blueprint) while allowing many agent identities in that tenant (one service principal per agent instance).

Unlike the classic pattern where every app instance might hold secrets, credentials are created and held only on the Agent Identity Blueprint, so individual agents do not need direct access to secrets—reducing risk. Great, right?

A simple analogy: the blueprint is the recipe, the blueprint principal is the licensed kitchen allowed to “cook” agents in your tenant, and each agent identity is a dish made from that recipe.

Foundry Agent Identity

You might not have noticed: every agent you develop and deploy on Foundry automatically gets an agent identity created in Entra ID.

In the following section, let’s run a hands-on demo: how to create Entra Agent ID objects and use them for authentication and authorization.

Demo

This demo uses PowerShell and the Microsoft Graph API. The steps are:

Step 1: Create the blueprint

1
2
3
4
5
6
7
8
9
10
11

$body = @{
    "@odata.type"         = "Microsoft.Graph.AgentIdentityBlueprint"
    displayName           = "ChrisBao Blueprint"
    "sponsors@odata.bind" = @("https://graph.microsoft.com/v1.0/users/$myUserId")
    "owners@odata.bind"   = @("https://graph.microsoft.com/v1.0/users/$myUserId")
}

$blueprint = Invoke-MgGraphRequest -Method POST `
    -Uri "https://graph.microsoft.com/beta/applications/" `
    -Headers @{ "OData-Version" = "4.0" } `
    -Body ($body | ConvertTo-Json)

The key parameter is Microsoft.Graph.AgentIdentityBlueprint, which tells Graph you are creating an Agent Identity Blueprint.

Step 2: Create the blueprint principal

1
2
3
4
5

$principalBody = @{ appId = $blueprint.appId }
$principal = Invoke-MgGraphRequest -Method POST `
    -Uri "https://graph.microsoft.com/beta/serviceprincipals/graph.agentIdentityBlueprintPrincipal" `
    -Headers @{ "OData-Version" = "4.0" } `
    -Body ($principalBody | ConvertTo-Json)

Step 3: Create credentials

1
2
3
4
5

$blueprintApp = (Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/beta/applications?`$filter=appId eq '$($blueprint.appId)'").value[0]
...
$secret = Invoke-MgGraphRequest -Method POST `
    -Uri "https://graph.microsoft.com/beta/applications/$($blueprintApp.id)/addPassword" `
    -Body ($secretBody | ConvertTo-Json)

As you can see, the secret is created under the blueprint, not on each agent instance.

Step 4: Get a blueprint token

1
2
3
4
5
6
7
8
9

$testTokenBody = @{
    client_id     = $blueprint.appId
    scope         = "https://graph.microsoft.com/.default"
    grant_type    = "client_credentials"
    client_secret = $secret.secretText
}

$testResponse = Invoke-RestMethod -Method POST `
    -Uri "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" `

An agent instance has no secret of its own and cannot authenticate by itself. We first use the blueprint’s secret to obtain a token. There is no user sign-in here—we use the OAuth 2.0 client credentials flow. If you want to go deeper on that flow, check my earlier articles on the topic.

Step 5: Validate token claims

1
2
3
4
5

$tokenPayload = Get-DecodedJwtToken -Token $blueprintToken | ConvertFrom-Json
if ($tokenPayload.roles -notcontains "AgentIdentity.CreateAsManager") {
    Write-Warning "  [WARN]  Token doesn't have AgentIdentity.CreateAsManager role. Waiting for permissions to propagate..."
    Start-Sleep -Seconds 10
}

The blueprint token must include the AgentIdentity.CreateAsManager role before you can create an agent identity.

Step 6: Create an agent identity

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$agentIdentityBody = @{
    displayName              = "My Agent"
    agentIdentityBlueprintId = $BlueprintAppId
    "sponsors@odata.bind"    = @("https://graph.microsoft.com/v1.0/users/$UserId")
}

$agentIdentity = Invoke-RestMethod -Method POST `
    -Uri "https://graph.microsoft.com/beta/serviceprincipals/Microsoft.Graph.AgentIdentity" `
    -Headers @{
        "Authorization" = "Bearer $blueprintToken"
        "OData-Version"   = "4.0"
        "Content-Type"    = "application/json"
    } `
    -Body ($agentIdentityBody | ConvertTo-Json)

With that token, we call Graph to create the agent identity.

Step 7: Exchange for an agent token

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# Exchange Blueprint token for agent token (Agent identity token)
$t2Body = @{
    client_id             = $AgentIdentityAppId
    scope                 = "https://graph.microsoft.com/.default"
    grant_type            = "client_credentials"
    client_assertion_type = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"
    client_assertion      = $blueprintToken
}

$t2Response = Invoke-RestMethod -Method POST `
    -Uri "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" `
    -ContentType "application/x-www-form-urlencoded" `
    -Body $t2Body

$agentToken = $t2Response.access_token
Write-Host "  [OK] Got T2 token (Agent identity)" -ForegroundColor Green

The agent needs its own token for API calls, so we exchange the blueprint token for an agent identity token using JWT bearer client assertion—no agent-held secret required for this step.

Step 8: Grant permissions to the agent identity

Even with an agent token, the identity has no effective permissions until app roles are assigned. Next we grant User.ReadWrite.All so the agent can read and write user data via Graph (adjust permissions to least privilege in production).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$permissionMap = @{
    "User.ReadWrite.All" = "741f803b-c850-494e-b5df-cde7c675a1ca"
}

$appRoleId = $permissionMap["User.ReadWrite.All"]

$permissionBody = @{
    principalId = $AgentIdentitySP
    resourceId  = $graphSP
    appRoleId   = $appRoleId
}
Invoke-MgGraphRequest -Method POST `
    -Uri "https://graph.microsoft.com/v1.0/servicePrincipals/$AgentIdentitySP/appRoleAssignments" `
    -Body ($permissionBody | ConvertTo-Json) | Out-Null
Write-Host "  [OK] Added permission: $permission" -ForegroundColor Green

Step 9: Verify the result

1
2
3
4
5
6

$response = Invoke-RestMethod -Method GET `
    -Uri "https://graph.microsoft.com/v1.0/users?`$top=5" `
    -Headers @{
        "Authorization" = "Bearer $AccessToken"
        "Content-Type"  = "application/json"
    }

The output shows the call succeeded.

1
2
3

Display Name User Principal Name                                 ID
------------ -------------------                                 --
Chris Bao    baoqger_gmail.com#EXT#@baoqgergmail.onmicrosoft.com d672a57e-86cc-46e3-b811-8fd26e0b0e50

Sidecar Architecture with Entra Agent Identity

Two common approaches to agent authentication both fall short:

• Hard-coded secrets in agent code. Every agent image holds a copy of your app’s client_secret. Any leak, log exposure, or committed .env can become a tenant-wide incident.
• Delegated user tokens for everything. The agent only acts when a human is present, and you lose per-agent auditability—every call can look like the same service principal.

As we saw in the demo, Entra Agent Identity handles authentication and authorization without embedding secrets in the agent and without requiring an interactive user for every action. So how do you use this in real applications?

The answer is the sidecar pattern: deploy a sidecar next to each agent that encapsulates Entra Agent Identity logic. The agent focuses on its job; the sidecar handles identity and tokens. I will unpack that architecture in a future article—stay tuned.

Summary

• Identity is the primary security perimeter—especially for autonomous agents calling APIs outside a fixed network edge.
• Entra ID uses app registration + service principal; Entra Agent ID adds blueprint, blueprint principal, and per-instance agent identity.
• Credentials stay on the blueprint; agents get distinct identities and tokens (including token exchange), which improves auditability and blast-radius control.
• Foundry can provision agent identities automatically; the PowerShell demo shows the same objects via Graph when you build custom agent hosts.

Background

In a previous article, I introduced Transaction Token and walked through how it supports fine-grained authorization. If you read that post, you probably felt the same thing I did: the approach is powerful, but it comes with real engineering cost.

In this article, I will share a lighter-weight OAuth extension—Rich Authorization Requests (RFC 9396)—that can also express fine-grained permissions. For AI Agents, that capability is especially valuable. After reading this post, you should understand why classic scope strings fall short, and how RAR adds structured authorization details without jumping straight to a full Transaction Token rollout.

In the OAuth 2.0 authorization flow, the user delegates access to an application within a bounded set of permissions. That boundary is carried in the authorization request as the scope parameter, for example:

1
2
3
4
5
6

GET /authorize?
   response_type=code
   &client_id=s6BhdRkqt3
   &state=af0ifjsldkj
   &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb
   &scope=openid%20profile%20email

Because this request is usually sent with GET, scope has to be a single string. That limits how much meaning and semantics you can pack into it.

Community practice often looks like this:

1

users:read users:write reports:read

The pattern is resource:action. When you need permissions on multiple resources, you separate them with spaces. That works well for requests like “let me read user profile data,” but it struggles with richer intent—for example, “let me transfer 45 to Merchant A.”

For AI Agents, fine-grained authorization is not a nice-to-have. Agents make autonomous decisions. If the consent screen cannot show those details, the user is effectively approving a blind spot—and that is dangerous. This gap between classic OAuth authorization and what agents need is one of the most common pain points teams hit when they wire agents into real APIs.

As we mentioned above, the Transaction Token path can record authorization scope inside the transaction context field (tctx). Smart design—but as I said at the start, the implementation cost is higher. In this post, let’s look at a lighter approach that is already on the standards track.

Rich Authorization Requests

Rich Authorization Requests (RAR) is an OAuth 2.0 extension standardized as RFC 9396. The idea is simple: add a new parameter, authorization_details, that describes authorization scope in a structured way.

To cover multiple resources or use cases, authorization_details is a JSON array. Each object describes one authorization scenario—for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

[
   {
      "type": "account_information",
      "actions": [
         "list_accounts",
         "read_balances",
         "read_transactions"
      ],
      "locations": [
         "https://example.com/accounts"
      ]
   },
   {
      "type": "payment_initiation",
      "actions": [
         "initiate",
         "status",
         "cancel"
      ],
      "locations": [
         "https://example.com/payments"
      ],
      "instructedAmount": {
         "currency": "EUR",
         "amount": "123.50"
      },
      "creditorName": "Merchant A",
      "creditorAccount": {
         "iban": "DE02100100109307118603"
      },
      "remittanceInformationUnstructured": "Ref Number Merchant"
   }
]

The type field identifies the resource or scenario. Your authorization server must register and support each type you use. Other fields depend on your business requirements.

When you send the authorization request, you URL-encode that JSON and attach it to the query string:

1
2
3
4
5
6

GET /authorize?
   response_type=code
   &client_id=s6BhdRkqt3
   &state=af0ifjsldkj
   &redirect_uri=https%3A%2F%2Fclient.example.org%2Fcb
   &authorization_details=%5B%7B%22type%22%3A%22account%5Finfo

From that angle, RAR is a remarkably straightforward way to move from coarse scope strings toward lightweight, fine-grained authorization—especially when you are prototyping agent workflows or comparing options before committing to a heavier Transaction Token architecture.

Summary

• Classic OAuth scope is a space-delimited string; it is easy to use but hard to express precise, transaction-level consent.
• AI Agents amplify that problem because users must understand what the agent is about to do, not just which API family it may call.
• Rich Authorization Requests (RFC 9396) add authorization_details so clients can describe structured permissions—including amounts, payees, and actions—in the authorization request itself.
• RAR does not replace every Transaction Token use case, but it is a lighter standards-based step toward fine-grained authorization. Great starting point, right?

Background

In this note, I’ll continue exploring one of the core capabilities of Azure AI Gateway: token-based rate limiting.

The hands-on setup is straightforward but very practical: I deploy a DeepSeek-R1 model on Microsoft Foundry, then integrate its online inference endpoint into Azure API Management (APIM). With APIM policies, we can enforce rate limits based on token consumption, so the “GenAI bill” doesn’t spiral out of control. Along the way, this is also a great way to revisit classic APIM concepts and apply them to modern GenAI workloads.

Why token-based rate limiting matters more for GenAI

For GenAI applications, rate limiting is not just about “getting an occasional 429 and retrying.” It directly affects:

• User experience: users can feel the latency or failures immediately when limits kick in
• Cost control: cost is strongly correlated with token usage; uncontrolled growth becomes expensive quickly
• System stability: token throughput spikes amplify pressure on upstream/downstream components (model capacity, gateway, logging pipeline, cache, etc.)

Compared to traditional APIs—where we typically limit by request rate (RPS/RPM) and burst traffic—GenAI rate limiting is often more nuanced:

• Platforms may limit both request count and token throughput
• Many platforms track prompt tokens and completion tokens separately
• Some platforms apply separate quotas for different modalities (text vs image vs audio)

That leads to an important reality: two requests are not equal in GenAI. The longer the context, the more content you stuff from RAG, the more complex your tool/function schema, and the longer the model’s answer, the faster tokens can explode. As a result:

• You might be under the request limit but still get throttled because tokens-per-minute is exhausted
• Or the other way around

Demo environment and services

To demonstrate token-based throttling, I set up a small lab environment with these Azure services:

• Azure API Management (APIM): exposes a unified API endpoint and hosts the throttling policies
• Azure Foundry: deploys DeepSeek-R1 and provides an online inference endpoint
• Log Analytics: the storage layer for token-related logs/metrics
• Application Insights: built on top of the Log Analytics workspace for querying and visualization

Architecture: the rate-limit design

The overall flow looks like this:

• Azure Foundry: deploy DeepSeek-R1 and expose its online inference endpoint
• APIM:
  • create a backend that points to the DeepSeek-R1 inference endpoint
  • create an api that wraps the backend, and expose it to end users and AI agents
  • implement and enforce rate-limit policies at the gateway layer
• Log Analytics: create a workspace to store token consumption data
• Application Insights: create an App Insights instance backed by the workspace for querying and dashboards

Deploying the DeepSeek-R1 model

Here is the DeepSeek-R1 deployment in Foundry:

Integrating APIM

After DeepSeek-R1 is deployed to Foundry, it exposes an inference endpoint. In APIM, I wrap it as a backend named foundry1:

Then I create an API named Inference API:

And bind it to the backend with the following policy snippet:

1

<set-backend-service backend-id="foundry1" />

This lab environment has quite a few moving parts, so I won’t expand every detail here. If you want the full working project, you can check my GitHub repo: azure-ai-gateway-labs.

Next, let’s focus on the key part: the rate-limit policy.

Rate-limit policies

The policy configuration looks like this:

In APIM, the two policies most relevant to token rate limiting are:

• llm-token-limit: enforce token-based throttling
• llm-emit-token-metric: emit token usage metrics (for logs/observability)

llm-token-limit: key configuration points

Here are the settings worth paying attention to:

• counter-key: the dimension used for token accounting. In this lab, I use subscription.Id.
  • subscription is an APIM concept for managing API access; clients can be required to include a subscription key (also commonly used as a lightweight security gate)
  • by extracting subscription info from the request, we can account and throttle per subscription
  • alternatively, you can throttle by client IP, e.g. Request.IpAddress
• tokens-per-minute: the core threshold, i.e. how many tokens are allowed per minute
  • you can also use token-quota + token-quota-period for more flexible quota periods (hour/day/month/year)
• estimate-prompt-tokens: whether to estimate tokens (prediction) or rely on the model’s usage returned from responses
  • when enabled, throttling may kick in more aggressively as you approach the limit
  • when disabled, it tends to be more conservative and accounts for actual consumption
• remaining-tokens-header-name, tokens-consumed-header-name: write remaining/consumed tokens into response headers for debugging (the demo makes this very obvious)

llm-emit-token-metric: observability and dimensions

llm-emit-token-metric collects token usage and sends it into an Application Insights custom namespace called llm.

Common metrics include:

• prompt tokens: input token count
• completion tokens: output token count
• total tokens: total consumption (sum of the two)

In addition, I attach two custom dimensions (tags) to each metric event: Client IP and API ID. This enables deeper filtering and analysis, for example:

Demo

Now let’s run a quick demo. The script below calls the APIM-wrapped DeepSeek-R1 inference endpoint 10 times and prints key response info so we can observe throttling behavior.

Note: variables like apim_resource_gateway_url, inference_api_path, inference_api_version, models_config, and apim_subscriptions come from your environment configuration. I keep them as-is.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

import json

import requests


def print_debug_headers(response):
    interesting_headers = [
        "x-apim-retry-after",
        "x-apim-remaining-tokens",
        "x-apim-tokens-consumed",
    ]

    debug_headers = {
        name: response.headers.get(name)
        for name in interesting_headers
        if response.headers.get(name) is not None
    }

    print("headers:", debug_headers if debug_headers else "<none>")


url = f"{apim_resource_gateway_url}/{inference_api_path}/models/chat/completions?api-version={inference_api_version}"

messages = {
    "messages": [
        {"role": "system", "content": "You are a sarcastic, unhelpful assistant."},
        {"role": "user", "content": "Can you tell me the time, please?"},
    ],
    "model": models_config[0]["name"],
}

api_runs = []

for i in range(10):
    response = requests.post(
        url, headers={"api-key": apim_subscriptions[0]["key"]}, json=messages
    )

    if response.status_code == 200:
        data = json.loads(response.text)

        input_tokens = data.get("usage").get("prompt_tokens")
        output_tokens = data.get("usage").get("completion_tokens")
        total_tokens = data.get("usage").get("total_tokens")

        print(
            "▶️ Run: ",
            i + 1,
            "status code: ",
            response.status_code,
            "✅",
            "input tokens: ",
            input_tokens,
            "output tokens: ",
            output_tokens,
            "total tokens: ",
            total_tokens,
        )

        print_debug_headers(response)
        # print("💬 ", data.get("choices")[0].get("message").get("content"))
    else:
        print("▶️ Run: ", i + 1, "status code: ", response.status_code, "⛔")
        print_debug_headers(response)
        print(response.text)

        total_tokens = 0

    api_runs.append((total_tokens, response.status_code))

The output below is the key observation. Since the limit is set to tokens-per-minute: 300:

• Run #1 consumes 258 tokens, leaving 42 remaining
• Run #2 still succeeds, but the remaining tokens drop to 0
• Subsequent runs get throttled with HTTP 429, and x-apim-retry-after tells you how long to wait before retrying

1
2
3
4
5
6
7
8
9
10

▶️ Run: 1 status code: 200 ✅ input tokens: 25 output tokens: 233 total tokens: 258 headers: {'x-apim-remaining-tokens': '42', 'x-apim-tokens-consumed': '258'}
▶️ Run: 2 status code: 200 ✅ input tokens: 25 output tokens: 184 total tokens: 209 headers: {'x-apim-remaining-tokens': '0', 'x-apim-tokens-consumed': '209'}
▶️ Run: 3 status code: 429 ⛔ headers: {'x-apim-retry-after': '28', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 28 seconds." }
▶️ Run: 4 status code: 429 ⛔ headers: {'x-apim-retry-after': '27', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 27 seconds." }
▶️ Run: 5 status code: 429 ⛔ headers: {'x-apim-retry-after': '25', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 25 seconds." }
▶️ Run: 6 status code: 429 ⛔ headers: {'x-apim-retry-after': '24', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 24 seconds." }
▶️ Run: 7 status code: 429 ⛔ headers: {'x-apim-retry-after': '22', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 22 seconds." }
▶️ Run: 8 status code: 429 ⛔ headers: {'x-apim-retry-after': '21', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 21 seconds." }
▶️ Run: 9 status code: 429 ⛔ headers: {'x-apim-retry-after': '19', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 19 seconds." }
▶️ Run: 10 status code: 429 ⛔ headers: {'x-apim-retry-after': '18', 'x-apim-remaining-tokens': '0'} { "statusCode": 429, "message": "Token limit is exceeded. Try again in 18 seconds." }

We can also validate this from logs:

Summary

In GenAI systems, token-based throttling is often more aligned with real resource consumption than request-count throttling. By using APIM’s llm-token-limit for quota enforcement and pairing it with llm-emit-token-metric to push token metrics into Application Insights / Log Analytics, we can manage cost, user experience, and stability under a single observable control plane.

In coming notes, I’d like to dig further into practical topics such as: designing quota tiers per subscription, handling burst traffic more gracefully, and building dashboards/alerts based on token usage. Great, right?

Background

In this article, I will continue the thread from my earlier post on the OAuth 2.0 Token Exchange authorization flow and dive into a more challenging piece of the puzzle: Transaction Token. We will look at why it exists, what problems it solves, and why those problems matter even more in the AI Agent era.

This post mixes big-picture security context with protocol details, so buckle up—it is a bit of a ride. After reading it, you should understand how txn-tokens fit into Zero Trust and multi-hop service calls.

The OAuth 2.0 Transaction Token Proposal

Transaction Token (txn-token for short) is still an Internet Draft. It comes from the OAuth working group, not a single author, and is already at draft version 11, which tells you how actively it is being reviewed and revised. It is not an RFC yet, but it has drawn a lot of attention in the community.

So what is the proposal trying to say? Let’s walk through it step by step.

Why Do We Need Transaction Tokens?

Modern systems are usually made of many services and apps. When a request enters your network from the Internet, it often fans out across several internal services. At that point, you have a few common ways to validate tokens.

Castle and moat

In this pattern, you validate the token at the network edge—typically the API gateway. Once that check passes, internal services call each other without repeating authentication or authorization.

That is the classic castle-and-moat model: the corporate network is the castle, and the perimeter is the moat. Outsiders cannot get in; but once you cross the boundary—say, after login or VPN access—users and devices are often treated as fully trusted and can move freely inside the castle.

Cloud and SaaS broke that model. The replacement many teams adopt is Zero Trust: never trust, always verify. Whether the caller is inside or outside the corporate network, every access request should be authenticated and authorized.

That is why teams moved toward the approach below.

Token Exchange

Service-to-service traffic still needs identity and permissions. A common shortcut is to reuse the user’s access token. But OAuth 2.0 was designed around an interactive flow while the user is present, granting the client a relatively static set of scopes up front. Passing the user token deep into the call graph greatly increases the risk of over-privileged leakage.

As I mentioned in the previous article, you can use the Token Exchange grant (RFC 8693): exchange the user token for one with a narrower, more appropriate scope, then pass that token to downstream services.

Even Token Exchange, however, leaves a meaningful attack surface:

• Spurious invocation: a compromised internal service calls downstream APIs when there is no real user-driven request.
• Identity impersonation: the caller crafts the request and pretends to be another user.
• Parameter tampering: a compromised middle tier changes fields in the JSON body (for example, order quantity).
• Token theft and replay: an attacker steals a bearer token from logs or memory and replays it until it expires.

Transaction Token is meant to remove—or at least shrink—that attack surface.

How Transaction Tokens Work

As the name suggests, a txn-token scopes authorization to a single transaction. “Transaction” here is not only a database transaction or a payment. It is a logical unit of work along one end-to-end business path. Because authorization is bound to that unit, txn-tokens are short-lived—often just a few minutes. You shrink the attack surface in time and in scope.

Next, let’s unpack the flow using a stock-trading example.

External call

The client sends a normal user access token to the API gateway:

1
2
3
4
5

POST /trade HTTP/1.1
Authorization: Bearer <oauth-access-token>
Content-Type: application/json

{ "action": "BUY", "ticker": "MSFT", "quantity": "100" }

Exchanging for a transaction token

The API gateway calls a Transaction Token Server (TTS). You can use the same interfaces and conventions as Token Exchange, because txn-token issuance is fundamentally another form of token exchange. In OAuth terms, the Authorization Server can act as the TTS.

1
2
3
4
5
6
7
8
9
10
11
12

POST /txn-token-service/token_endpoint HTTP/1.1
Host: txn-token-service.trust-domain.example
Content-Type: application/x-www-form-urlencoded

grant_type=urn:ietf:params:oauth:grant-type:token-exchange
&requested_token_type=urn:ietf:params:oauth:token-type:txn_token
&audience=trust-domain.example
&scope=trade.stocks
&subject_token=eyJhbGciOiJFUzI1NiIsImtpZC...
&subject_token_type=urn:ietf:params:oauth:token-type:access_token
&request_context={"req_ip":"69.151.72.123","authn":"urn:ietf:rfc:6749"}
&request_details={"action":"BUY","ticker":"MSFT","quantity":"100"}

The exchange carries context and details via request_context and request_details. Notice that audience is no longer a single downstream API—it represents the trust boundary for this transaction.

What is inside the transaction token?

The txn-token is a signed JWT. The TTS derives claims from the subject token (the user access token) and from the request, including request_details and request_context, and adds further attributes as policy allows.

A typical txn-token header looks like this:

1
2
3
4
5

{
  "typ": "txntoken+jwt",
  "alg": "RS256",
  "kid": "tts-2026-q2"
}

The type is txntoken+jwt. kid identifies the public key downstream services use to verify the signature.

The payload might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

{
  "iat": 1686536226,
  "aud": "trust-domain.example",
  "exp": 1686536586,
  "txn": "97053963-771d-49cc-a4e3-20aad399c312",
  "sub": "d084sdrt234fsaw34tr23t",
  "req_wl": "apigateway.trust-domain.example",
  "scope": "trade.stocks",
  "rctx": {
    "req_ip": "69.151.72.123",
    "authn": "urn:ietf:rfc:6749"
  },
  "tctx": {
    "action": "BUY",
    "ticker": "MSFT",
    "quantity": "100",
    "customer_type": { "geo": "US", "level": "VIP" }
  }
}

Key fields:

• sub: user identifier, taken from the subject token.
• aud: the trust boundary; the token is only meaningful inside that boundary.
• exp: short lifetime—on the order of minutes.
• txn: unique id for this transaction.
• req_wl: who initiated the exchange (often the gateway).
• scope: same idea as in ordinary access tokens.
• rctx: request context (for example, client IP).
• tctx: transaction context. Important: the TTS should not blindly copy the request body into tctx; that would be easy to abuse. The TTS decides what may be bound into tctx.

The TTS signs the JWT with its private key and returns it as access_token in the Token Exchange response—exactly as RFC 8693 defines.

1
2
3
4
5
6
7
8
9

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

{
  "token_type": "N_A",
  "issued_token_type": "urn:ietf:params:oauth:token-type:txn_token",
  "access_token": "eyJ0eXAiOiJ0eG50b2tlbitqd3QiLCJhbGciOiJSUzI1NiIsImtpZCI6InR0cy0yMDI2LXEyIn0..."
}

Propagation and verification

Downstream calls should send the txn-token in the Txn-Token HTTP header, because Authorization is reserved for ordinary access tokens:

1
2
3
4
5
6

POST /orders HTTP/1.1
Host: order-service.trust-domain.example
Txn-Token: eyJ0eXAiOiJ0eG50b2tlbitqd3QiLC...
Content-Type: application/json

{"action": "BUY", "ticker": "MSFT", "quantity": 100}

The downstream service typically verifies:

• Signature: validate with the TTS public key so the payload cannot be altered.
• aud: matches this deployment’s trust boundary (usually configuration).
• exp: token is still valid; lifetimes are short by design.
• tctx: this is the txn-token superpower. A compromised service might change the body from “buy 100 shares” to “buy 1000 shares,” but tctx is inside the signed JWT—downstream code should trust tctx, not the mutable body. Smart design, right?

What Transaction Tokens Buy You

Now that we have walked through the flow, let’s map txn-tokens back to the four attack patterns:

• Spurious invocation: only the TTS can mint a valid txn-token. A compromised service cannot forge one; downstream verification fails fast.
• User impersonation: sub is part of the signed JWT and cannot be rewritten in transit.
• Parameter tampering: business parameters live in tctx. Verifiers read authoritative data from the token, not from attacker-controlled JSON.
• Token theft and replay: txn-tokens are tied to a single audience (one trust boundary), carry minimal scope, expire quickly, and include a unique txn id so you can enforce one-time use with a cache. Even if theft happens, the window is tiny.

Agentic AI

Rolling out transaction tokens is real engineering work—it can feel heavy for classic three-tier apps. For agentic AI systems, though, where tool calls, delegation, and multi-hop workflows are the norm and behavior is harder to predict, tightening the authorization attack surface is often worth the investment.

Background

In my previous two articles, I walked through edge AI development on the .NET stack using the ONNX model format. Those posts relied on CPU hardware. In this article, I will extend that path and use a GPU for GenAI inference. If you want to know how much faster inference can get on GPU—or how to set up a local CUDA environment on Windows—this post is for you. Let’s get started.

Model

As I mentioned in the first article in this series, the Phi-3-Mini-4K-Instruct-onnx model ships in several variants for different compute targets: cuda, directml, and cpu_and_mobile.

In that earlier post we used the cpu_and_mobile build. Today we switch to the cuda folder. We still use a quantized model; the weight file is about 2.3 GB.

The model’s genai_config.json makes the execution provider explicit—provider_options is set to CUDA:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

{
    "model": {
        "bos_token_id": 1,
        "context_length": 4096,
        "decoder": {
            "session_options": {
                "log_id": "onnxruntime-genai",
                "provider_options": [
                    {
                        "cuda": {
                            "enable_cuda_graph": "0"
                        }
                    }
                ]
            },
            "filename": "phi3-mini-4k-instruct-cuda-int4-rtn-block-32.onnx",
            "head_size": 96,
            "hidden_size": 3072,
            "inputs": {
                "input_ids": "input_ids",
                "attention_mask": "attention_mask",
                "past_key_names": "past_key_values.%d.key",
                "past_value_names": "past_key_values.%d.value"
            },
            "outputs": {
                "logits": "logits",
                "present_key_names": "present.%d.key",
                "present_value_names": "present.%d.value"
            },
            "num_attention_heads": 32,
            "num_hidden_layers": 32,
            "num_key_value_heads": 32
        },
        "eos_token_id": [
            32000,
            32001,
            32007
        ],
        "pad_token_id": 32000,
        "type": "phi3",
        "vocab_size": 32064
    },
    "search": {
        "diversity_penalty": 0.0,
        "do_sample": false,
        "early_stopping": true,
        "length_penalty": 1.0,
        "max_length": 4096,
        "min_length": 0,
        "no_repeat_ngram_size": 0,
        "num_beams": 1,
        "num_return_sequences": 1,
        "past_present_share_buffer": true,
        "repetition_penalty": 1.0,
        "temperature": 1.0,
        "top_k": 1,
        "top_p": 1.0
    }
}

Next step, let’s look at the hardware I used for this demo.

GPU hardware

My machine has an NVIDIA RTX PRO 4000 Blackwell Generation Laptop GPU:

It has 16 GB of VRAM—enough headroom for small models like Phi-3-Mini-4K-Instruct-onnx. Great, right?

Environment

To run this ONNX model on GPU with CUDA, you need four layers of software on top of the hardware:

• NVIDIA driver — required for any GPU workload; nothing special to add here.
• CUDA Toolkit — the CUDA SDK (compiler, runtime, debugger, and related tools). Pick a toolkit version that matches your GPU generation. On my Blackwell laptop I use CUDA 13.4. Installing the toolkit takes time, so confirm your GPU architecture and the ORT/GenAI package requirements before you install.
• cuDNN — NVIDIA’s deep learning primitives library; ONNX Runtime GPU depends on it.
• Microsoft.ML.OnnxRuntimeGenAI.Cuda — the .NET package for CUDA. It pulls in native dependencies such as ONNX Runtime GPU. Package version must align with your CUDA stack—for example, 0.16.0 targets the CUDA 13 line.

Setting this up involves trial and error; give yourself some patience. Once everything is in place, we can move to the demo.

Demo

Switching from CPU to GPU does not require rewriting your agent. You mainly change how the model is loaded.

Lower-level API:

1
2
3
4
5
6
7
8
9

using Microsoft.ML.OnnxRuntimeGenAI;

string modelPath = @"C:\path\to\cuda-model";
using Config config = new Config(modelPath);

config.ClearProviders();
config.AppendProvider("cuda");

using Model model = new Model(config);

Higher-level API (Microsoft.Extensions.AI):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

using Microsoft.Extensions.AI;
using Microsoft.ML.OnnxRuntimeGenAI;

string modelPath = @"C:\path\to\cuda-model";

using Config config = new Config(modelPath);
config.ClearProviders();
config.AppendProvider("cuda");

using var client = new OnnxRuntimeGenAIChatClient(config);

ChatResponse response = await client.GetResponseAsync("Hello", new ChatOptions
{
    MaxOutputTokens = 64,
});

The important line is AppendProvider("cuda").

As we mentioned above, the model folder must be the cuda variant (not cpu_and_mobile). The config snippet simply tells ONNX Runtime GenAI to use the CUDA execution provider at runtime.

CPU vs GPU throughput

First, the CPU run:

438 tokens in 25 seconds — about 17 tokens/s.

Then the GPU run:

Same 438 tokens in 3.5 seconds — about 121 tokens/s. That is roughly a 7–8× speedup. Smart design, right? Most of the win comes from moving matmul-heavy work off the CPU and onto the GPU execution provider.

Summary

After reading this post, you should understand:

• Why the Hugging Face cuda ONNX build pairs with OnnxRuntimeGenAI.Cuda on .NET.
• The four-layer stack: driver → CUDA Toolkit → cuDNN → GenAI CUDA NuGet—and why versions must match (especially on newer GPUs such as Blackwell).
• How little code changes when you switch providers: ClearProviders() + AppendProvider("cuda").
• What kind of speedup to expect on a small quantized Phi-3 model when CPU and GPU are compared fairly.

OAuth 2.0 was designed in an era where the main security question was straightforward: how can a user authorize an application to access a protected resource? That model still works well for traditional web apps and APIs, but AI agents introduce a new kind of actor into the system. An AI agent is not just a browser, not just a backend service, and not merely a normal client application. It is a decision-making component that can interpret user goals, choose actions, and call tools or APIs with a degree of autonomy. That shift matters for authorization.

To understand why a new OAuth extension is needed for AI agents, it helps to start with the two existing patterns: the standard Authorization Code Flow and OAuth Token Exchange. Both solve important problems. But neither fully captures what it means for a specific AI agent to act on behalf of a user.

1. What the Standard Authorization Code Flow Solves

The standard Authorization Code Flow for a server-side web app is built around a simple trust model.

A user is redirected to the authorization server, signs in, and grants consent. The authorization server then returns an authorization code to the application. The backend redeems that code using its client_id and client_secret, and receives tokens. The basic meaning of this flow is clear:

• the user approved access
• the application proved its identity as the registered client
• the authorization server issued tokens based on that combination

This is a strong design for classic web applications because it binds user approval to a trusted confidential client. It also protects the token exchange step by requiring the backend to authenticate itself with a client_secret or similar credential.

However, this model only really understands two important parties:

• the user
• the client application

That was enough when the client application was assumed to be the only meaningful actor. But in an AI system, that assumption starts to break down.

2. Why the Standard Authorization Code Flow Is Not Enough for AI Agents

Inside an AI application, there may be multiple internal agents with very different behaviors and risk profiles. One agent may summarize emails. Another may send messages. Another may create financial reports or call external systems. From a security point of view, those are not all the same thing.

But standard Authorization Code Flow has no native way to express that distinction.

When the user consents in the classic flow, the consent screen can say:

• this application wants access
• these scopes are requested

What it cannot clearly say is:

• this specific agent inside the app wants to act for you
• this agent may read your calendar but not send email
• this agent is the one that will perform the next step autonomously

In other words, the standard flow gives consent to the client, not to the internal actor. That creates a visibility gap. The authorization server knows which app received consent, but it does not know which agent inside that app is actually going to act. The resource server later sees a token, but it may only know the user and the client, not the specific agent making decisions.

That gap becomes important because AI agents are not passive code paths. They can choose actions dynamically. They can chain tools together. They can operate across multiple hops. If the protocol cannot identify the acting agent, then user delegation becomes too coarse. The user is effectively authorizing the whole application, even if only one specific agent should be trusted for a given task.

Fixing that gap might require richer delegation in the token, not just a better consent screen — which is why Token Exchange is often discussed alongside agents.

3. What OAuth Token Exchange Solves

OAuth Token Exchange, described in RFC 8693, addresses a similar problem in micro service architectures.

It starts from the idea that one token is often the wrong token for the next hop. A frontend service may receive a user token, but a downstream API may need a different token with a different audience, narrower scope, or explicit delegation context. Instead of blindly forwarding the original token, the service can exchange it for a new one better suited for the downstream service.

This is valuable for several reasons:

• it reduces scope
• it enforces correct audience boundaries
• it improves trust separation between services
• it can preserve delegation semantics using actor-related claims such as act

This is already much closer to the AI-agent problem than classic Authorization Code Flow is. Token Exchange understands that the entity presenting a token and the subject represented by that token may not be the same party. It gives us a more precise model for multi-hop systems.

That is why Token Exchange feels like an important building block for agent security.

4. Why Token Exchange Still Is Not Enough for AI Agents

Even though Token Exchange is closer to what AI agents need, it still does not fully solve the problem.

The biggest reason is that Token Exchange is mainly a back-channel mechanism. It assumes the calling service already has some token or authority and now wants to transform it into a different token for another context. That works well in service-to-service architectures. But AI-agent delegation often needs something more: explicit user consent in the front channel.

An AI agent may decide that it now needs permission to access a new resource or perform a higher-risk action. At that moment, the system should not simply assume that the client can translate existing authority into agent authority. The user may need to be asked directly:

• do you allow this specific agent to act for you?
• do you allow it for these scopes?
• do you allow it in this context?

Plain Token Exchange does not define that consent step. It helps you transform authority, but it does not define how the user approves a particular internal agent in the first place.

There is also another limitation: Token Exchange can model delegation, but it does not make the AI agent a first-class identity in the end-to-end flow. In many AI systems, the real security question is not just “does the application have permission?” but “which exact agent is exercising that permission right now?” If that identity is not explicitly captured and verified, then the protocol cannot strongly bind user approval to the acting agent.

5. The Core Problem: AI Agents Create a New Security Boundary

Traditional OAuth assumes the client application is the relevant unit of trust. In an AI system, that is no longer always true.

The client is still important, but now there is a second boundary inside the client:

• the client is the application boundary
• the agent is the decision-making boundary

That distinction matters because the agent is often the component that interprets intent and chooses actions. It is the one deciding whether to query a calendar, send a message, call a database, or invoke another tool. If the security system only sees the outer client, it misses the actual actor driving the behavior.

This creates several risks:

• the user may consent to an application without understanding which agent will act
• one agent may misuse permissions intended for another agent
• the authorization server may be unable to verify that the same agent approved in the consent step is the one redeeming the delegation
• the resource server may be unable to audit which specific agent performed the action

In short, AI agents introduce a new layer of autonomy, and that autonomy needs to become visible to the authorization system.

6. What the New OAuth Extension Adds

Although OAuth has not yet published a formal RFC for AI-agent authorization, the OAuth working group and individual developers have already contributed a number of approaches. This article walks through one such proposal as an example; other drafts and designs are worth exploring separately.

That proposal is the draft OAuth 2.0 Extension: On-Behalf-Of User Authorization for AI Agents. https://www.ietf.org/archive/id/draft-oauth-ai-agents-on-behalf-of-user-01.html

The new OAuth extension for AI agents addresses this by combining the strengths of the two older models.

From Authorization Code Flow, it keeps the front-channel consent model. The user is still redirected to the authorization server and can still approve access interactively.

From Token Exchange, it borrows the delegation mindset. The final token should not merely represent “the app acting for the user.” It should represent “this specific agent acting for the user through this client.”

To make that work, the extension adds new protocol elements such as:

• a requested_actor parameter in the authorization request
• an actor_token in the token request
• an act claim in the resulting access token

These additions matter because they create a full chain of authorization meaning:

1. The client tells the authorization server which agent is requesting delegation.
2. The user consents not only to the app and scope, but also to that specific agent.
3. The agent proves its identity later using an actor_token.
4. The authorization server verifies that the actor redeeming the flow is the same actor the user approved.
5. The resulting access token records both the user and the agent.

That closes the gap that exists in both older flows.

7. Why This Is Better Than Reusing the Old Flows As-Is

Without the new extension, a system would have to improvise.

If it used only Authorization Code Flow, it could authenticate the app and capture user consent, but it would not clearly identify the internal agent. The user’s approval would be too broad.

If it used only Token Exchange, it could model delegation better, but it would miss the front-channel consent experience needed for user-approved agent behavior. The protocol would know how to transform authority, but not how to get the user’s explicit approval for a specific agent.

The new extension is needed because AI-agent security requires both:

• explicit user consent
• explicit actor identity

Standard Authorization Code Flow gives you the first, but not the second. Token Exchange gives you pieces of the second, but not the first in the right form. The extension combines them into a model that fits agent systems.

8. The Bigger Security Meaning

At a deeper level, this is not just a protocol convenience. It is a shift in security thinking.

Older OAuth patterns mostly answer the question: can this application access this resource for this user?

Agent-based systems force us to ask a more precise question: which autonomous actor inside the application is acting for the user, and did the user explicitly approve that delegation?

That is a more demanding requirement, but it matches the reality of AI systems. AI agents are not just UI features. They are operational actors. They need explicit identity, explicit delegation, and explicit auditability.

Conclusion

The new OAuth extension for AI agents is needed because AI changes the unit of trust.

Standard Authorization Code Flow assumes the client application is the meaningful actor. Token Exchange assumes delegated authority can be transformed across service boundaries. But AI agents sit between those two models: they need fresh user consent like an interactive client flow, and they need explicit delegation semantics like a multi-hop service flow.

That is why a new extension makes sense. It allows OAuth to move from a world where “an app gets a token for a user” to a world where “a specific agent is delegated limited authority to act for a user, and every party in the system can see and verify that fact.”

Background

In a previous article, I walked through the three most common OAuth 2.0 grant flows: Authorization Code Flow, PKCE Flow, and Device Flow. We looked at which scenarios and application types each one fits, what problems they solve, and how they differ in implementation.

In this post, I will add another important flow: Token Exchange. After reading it, you should have a clearer picture of how token exchange works—and why that understanding pays off when you start thinking about AI Agent authorization. Let’s get started.

The Token Exchange (RFC 8693) flow is straightforward at a high level: when a client already holds a token, it asks the authorization server (Authorization Server) to issue a new token that is better suited to another service or another security context. You exchange an old token for a new one—hence Token Exchange.

Why Do We Need Token Exchange?

Modern systems are usually made of many services and applications. Token Exchange is especially valuable for service-to-service communication and cross-boundary scenarios. Suppose an upstream API receives the user’s token and then needs to call a downstream API. Instead of forwarding the original user token everywhere, the upstream API can exchange it for a new token that is meant only for the downstream API. That pattern helps in several ways:

• Matching audience: A user token is often issued for one specific API. The receiver should align with the token’s aud (audience). Otherwise you are accepting a token that was not minted for you—a classic trust boundary problem.
• Appropriate scope: The original user token may carry permissions intended for the first API. If every downstream API keeps using that same token, they often end up with more authority than they need—the least privilege problem.
• Explicit actor: User tokens usually do not carry delegation metadata. Downstream services often need to know who delegated access (delegator) and who is actually performing the action (actor).
• Controlled token forwarding: If every hop only forwards the original user token—for example, browser → API gateway → service A → service B → service C—then any compromise of one service exposes a token that still works elsewhere. That is the blast radius problem.

So the concerns Token Exchange addresses line up closely with how we think about network security:

• Least privilege: Every component gets only the access it needs.
• Trust boundary: A token issued for Service A should not be automatically trusted by Service B.
• Context: Do not only ask “Is this token valid?”—also ask “Valid for whom? For what?”
• Defense in depth: Even if one token leaks, limit where it can be used.
• Blast radius reduction: Keep token compromise as local as possible instead of spreading across the whole system.
• Auditability and attribution: Know who the user is and which service is acting.
• Explicit delegation: Make delegation visible; do not let one party silently inherit another’s rights.

Unlike other OAuth flows, Token Exchange centers on a concept that matters a lot for AI agents: delegation. That is why I see it as groundwork for agent authorization.

Impersonation vs Delegation

The Token Exchange specification spends considerable effort distinguishing impersonation from delegation. Let’s walk through a concrete example.

Assume this setup:

• User: Alice
• Frontend service: orders-web
• Backend service: inventory-api
• Authorization server: auth.example.com

Alice uses the frontend. orders-web holds Alice’s token but needs a new token scoped for inventory-api. Token Exchange covers two patterns: impersonation and delegation.

Impersonation

In the impersonation case, orders-web requests a token that lets it act as Alice when calling inventory-api:

1
2
3
4
5
6
7
8
9

POST /token HTTP/1.1
Host: auth.example.com
Content-Type: application/x-www-form-urlencoded
Authorization: Basic b3JkZXJzLXdlYjpzZWNyZXQ=

grant_type=urn:ietf:params:oauth:grant-type:token-exchange&
subject_token=user-access-token&
subject_token_type=urn:ietf:params:oauth:token-type:access_token&
audience=inventory-api

subject_token is the subject’s token—in our story, Alice’s original user token. That identity shows up in the new token’s sub claim. audience is the intended recipient API, here inventory-api.

Key claims in the new token:

1
2
3
4
5

{
  "sub": "alice",
  "aud": "inventory-api",
  "scope": "inventory.read"
}

From inventory-api‘s perspective, the token is about Alice. It has no signal that orders-web is the caller. That is why this pattern is called impersonation.

Delegation

In the delegation case, orders-web sends a slightly richer request:

1
2
3
4
5
6
7
8
9
10
11

POST /token HTTP/1.1
Host: auth.example.com
Content-Type: application/x-www-form-urlencoded
Authorization: Basic b3JkZXJzLXdlYjpzZWNyZXQ=

grant_type=urn:ietf:params:oauth:grant-type:token-exchange&
subject_token=user-access-token&
subject_token_type=urn:ietf:params:oauth:token-type:access_token&
actor_token=orders-web-service-token&
actor_token_type=urn:ietf:params:oauth:token-type:access_token&
audience=inventory-api

Besides subject_token, the client also supplies actor_token—the token for the acting party, orders-web. That relationship appears in the act claim:

1
2
3
4
5
6
7
8

{
  "sub": "alice",
  "aud": "inventory-api",
  "scope": "inventory.read",
  "act": {
    "sub": "orders-web"
  }
}

Now inventory-api can see: the request concerns Alice, and orders-web is the actor performing the call on her behalf. That is delegation.

Great—once delegation is clear, the next question is natural: what does this mean for AI Agent authorization?

AI Agent Scenarios

Compared with traditional apps, AI agents stand out because of autonomy: an agent can decide its own next steps. One client application may host multiple agents, each able to act on its own. For AI agents, an explicit delegation chain is not a nice-to-have—it is often required. That is a big reason many agent authorization designs lean on Token Exchange.

For example, AWS AgentCore and Azure Foundry Agent both support On-Behalf-Of Token Exchange-style authorization. I will dig into how those implementations work in upcoming posts—stay tuned!

Background

In my previous article, I walked through how to load a small language model locally and build an AI app on top of the cross-platform ONNX model standard. That post focused on the low-level APIs exposed by ONNX Runtime GenAI — great for studying and practicing how large models work under the hood, but a bit too low-level for day-to-day Agent development.

In this article, I will share what I learned about how the .NET ecosystem supports AI Agents today, and how you can plug ONNX into that stack without staying at the Generator / tokenizer loop level. After reading this post, you should understand where Microsoft.Extensions.AI (MEAI) fits, how OnnxRuntimeGenAIChatClient simplifies local chat, and how to take one more step with Microsoft Agent Framework (MAF) for a minimal on-device Agent.

If you have already been through the low-level ONNX GenAI walkthrough, this post is the natural next step. Let’s get started.

First, the Microsoft.ML.OnnxRuntimeGenAI package exposes a higher-level type: OnnxRuntimeGenAIChatClient. With it, we can talk to a local small model much more easily and skip most of the low-level GenAI API details.

If you look at the implementation, OnnxRuntimeGenAIChatClient is built on the IChatClient interface — and that leads us to an important piece of the .NET AI stack: Microsoft.Extensions.AI (MEAI).

MEAI is a set of core .NET libraries that provide a unified layer of C# abstractions for AI services: small and large language models, embeddings, middleware, and more. Low-level APIs such as IChatClient were extracted from Semantic Kernel and are now part of MEAI. The goal is to act as a unifying layer in the .NET ecosystem so you can pick your preferred frameworks while still sharing the same core concepts across libraries.

Next, let’s see what an app looks like when we combine MEAI with ONNX Runtime GenAI:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

using Microsoft.Extensions.AI;
using Microsoft.ML.OnnxRuntimeGenAI;

var modelPath = "path-2-onnx-model";

using var client = new OnnxRuntimeGenAIChatClient(modelPath, new()
{
    EnableCaching = false,
});

var messages = new List<ChatMessage>
{
    new(ChatRole.System, "You are a helpful AI assistant."),
    new(ChatRole.User, "hi"),
};

ChatResponse response = await client.GetResponseAsync(
    messages,
    new ChatOptions
    {
        Temperature = 0.7f,
        TopP = 0.9f,
        AdditionalProperties = new()
        {
            ["num_beams"] = 1,
            ["do_sample"] = true,
        },
    });

Console.WriteLine(response.Text);

As you can see, besides OnnxRuntimeGenAIChatClient, we also use types such as ChatMessage and ChatOptions from Microsoft.Extensions.AI. Generation options like num_beams and do_sample can be passed through AdditionalProperties when you need GenAI-specific knobs — handy, right?

One more distinction worth keeping in mind: MEAI alone is not an Agent framework. You can build one-shot calls, chat, or even tool-call loops on top of MEAI without the system becoming fully “agentic.” When you need goal-directed, multi-step orchestration, that is when you reach for Microsoft Agent Framework (MAF) instead.

In the following section, let’s take that next step and wire a local ONNX small model into a minimal edge Agent.

Microsoft Agent Framework (MAF)

The jump from chat client to Agent is surprisingly small: with the AsAIAgent extension on IChatClient, you can promote a chatbot into an AIAgent with instructions and a name.

1
2
3
4
5
6
7
8
9
10
11
12

using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using Microsoft.ML.OnnxRuntimeGenAI;

var modelPath = "path-2-onnx-model";

// Get a chat client for ONNX and use it to construct an AIAgent.
using OnnxRuntimeGenAIChatClient chatClient = new(modelPath);
AIAgent agent = chatClient.AsAIAgent(instructions: "You are good at telling jokes.", name: "Joker");

// Invoke the agent and output the text result.
Console.WriteLine(await agent.RunAsync("Tell me a joke about a pirate."));

Same model path, same ONNX runtime underneath — but now you are in MAF’s Agent surface area for orchestration and tooling as your scenarios grow.

Summary

In this post, we connected three layers:

1. ONNX Runtime GenAI — local inference for small models on device
2. MEAI (IChatClient, ChatMessage, ChatOptions) — a clean, shared abstraction for model interaction in .NET
3. MAF (AsAIAgent) — when chat is not enough and you need Agent-style orchestration

Based on the two examples above, I put together a map of the ONNX-related .NET ecosystem you can refer to when you design your own edge AI or Agent projects:

As we mentioned above, the low-level GenAI APIs are still the best classroom for understanding transformers and autoregressive loops. For product-style Agent work on .NET, this higher-level path is the one I reach for first.

Background

In my previous articles, I have shared quite a bit about RAG systems. One of the most important components in any RAG pipeline is vector storage. High-performance vector databases are usually expensive. But what if you need to store massive amounts of vector data, and latency is not your top priority? Can we find a balance between performance and cost? And if so, what storage option fits that trade-off?

That is exactly where AWS S3 Vector comes in. In this article, I will share what I learned about this service and walk you through a hands-on example.

After reading this post, you should understand how S3 Vector extends the S3 object storage model to vector data, and how to use it in a simple RAG-style workflow.

From S3 to S3 Vector

Characteristics of S3 Object Storage

Among the three major storage types — object storage, block storage, and file storage — S3 is the classic example of object storage. Unlike the other two, S3 stores data as objects inside a flat container (in S3, this container is called a Bucket). Each object contains a key (name), data content, and metadata. The structure looks roughly like this:

1
2

Bucket
  └── Object (key → file/blob + metadata)

Because of this design, S3 offers several advantages:

• Extremely strong scalability
• Very low cost for massive, infrequently modified data

Of course, it also has limitations, such as:

• Not ideal for frequent random read/write operations — you work at the whole-object level
• Relatively higher latency

That is why AWS S3 (or Azure Blob Storage) is a great fit for backups, media files, logs, and other static data scenarios. In more complex architectures, S3 often works alongside high-performance storage services to form a hot/cold multi-tier storage design.

In the AI era, this same pattern naturally extends to vector storage.

How S3 Vector Extends S3

From a product positioning perspective, S3 Vector solves the same problem that S3 solved in the traditional cloud era: store more data at a lower cost. The difference is that the data being stored is vector data.

Of course, you could store raw semantic vectors directly in a classic S3 Bucket. But for AI applications, raw vector data alone is not very useful. What we actually need is similarity search over vectors — the foundation for building Agents or RAG systems.

So S3 Vector keeps the familiar Bucket concept and introduces a new storage structure on top of it:

1
2
3

Vector bucket
  └── Vector index
        └── Vector (key → embedding + metadata)

The key difference is that S3 Vector introduces an index layer, where embeddings and metadata are stored as key-value pairs.

You can still read an embedding by key, but in real AI applications, what you usually need is to search for information that is semantically closest to a query.

That is the biggest difference from a classic S3 Bucket. Beyond storage, S3 Vector also manages underlying compute resources in a Serverless way. When a user query comes in, the service performs the search for you.

This Serverless, pay-as-you-go compute model means resources are triggered only when APIs are accessed — not kept running while idle. That keeps compute costs as low as possible.

Smart design, right? It is a strong fit for scenarios with massive vector storage, moderate access volume, and relaxed latency requirements — very much aligned with the design philosophy of object storage.

Next, let’s walk through a simple example to see how S3 Vector works in practice.

Demo

Create a Vector Bucket

As you can see, S3 Vector has its own dedicated client:

1
2
3
4
5
6
7
8
9
10
11

import boto3

session = boto3.Session()
region = session.region_name

# Create a S3 Vectors client in the AWS Region of your choice.
s3vectors = boto3.client("s3vectors", region_name=region)
bedrock = boto3.client("bedrock-runtime", region_name=region)

# Create a vector bucket
s3vectors.create_vector_bucket(vectorBucketName="media-embeddings")

The created Vector Bucket looks like this:

Create a Vector Index

Next, let’s create a Vector Index inside the Vector Bucket:

1
2
3
4
5
6
7
8
9

# Create a vector index "movies" in the vector bucket "media-embeddings" with non-filterable metadata keys
s3vectors.create_index(
    vectorBucketName="media-embeddings",
    indexName="movies",
    dimension=1024,
    distanceMetric="cosine",
    dataType="float32",
    metadataConfiguration={"nonFilterableMetadataKeys": ["source_text"]}
)

• dimension and dataType define the vector dimension and numeric type. These must match the embedding model you use later.
• distanceMetric defines how vector distance is calculated. Since we are working with plain text in this article, cosine is a good choice.
• metadataConfiguration controls metadata behavior. By default, metadata fields support filtering. But some fields are not ideal for filtering — for example, the original source text. That field is mainly there to provide context so you know what the search result actually is. So we mark source_text as non-filterable.

The Vector Index details look like this:

Embedding and Data Ingestion

Now we can ingest data. First, we choose amazon.titan-embed-text-v2:0 as our embedding model.

Then we call put_vectors to write embedding data into the index:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

import json

# Texts to convert to embeddings.
texts = [
    "Star Wars: A farm boy joins rebels to fight an evil empire in space",
    "Jurassic Park: Scientists create dinosaurs in a theme park that goes wrong",
    "Finding Nemo: A father fish searches the ocean to find his lost son"
]

# Generate vector embeddings.
embeddings = []
for text in texts:
    response = bedrock.invoke_model(
        modelId="amazon.titan-embed-text-v2:0",
        body=json.dumps({"inputText": text})
    )

    # Extract embedding from response.
    response_body = json.loads(response["body"].read())
    embeddings.append(response_body["embedding"])

# Write embeddings into vector index with metadata.
s3vectors.put_vectors(
    vectorBucketName="media-embeddings",
    indexName="movies",
    vectors=[
        {
            "key": "Star Wars",
            "data": {"float32": embeddings[0]},
            "metadata": {"source_text": texts[0], "genre": "scifi"}
        },
        {
            "key": "Jurassic Park",
            "data": {"float32": embeddings[1]},
            "metadata": {"source_text": texts[1], "genre": "scifi"}
        },
        {
            "key": "Finding Nemo",
            "data": {"float32": embeddings[2]},
            "metadata": {"source_text": texts[2], "genre": "family"}
        }
    ]
)

Pay attention to the structure of each vector entry:

1
2
3
4
5

{
    "key": "xxxx",
    "data": {"float32": [0.12, -0.34, ...]},
    "metadata": {"source_text": "xxxx", "genre": "xxx"}
}

Notice that float32 matches the type we defined when creating the index. The source_text field will not support filtering, while other metadata fields will.

Query and Search

As mentioned above, S3 Vector is more often used for similarity search via query_vectors, rather than reading an embedding by key.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Query text to convert to an embedding.
input_text = "adventures in space"

# Generate the vector embedding.
response = bedrock.invoke_model(
    modelId="amazon.titan-embed-text-v2:0",
    body=json.dumps({"inputText": input_text})
)

# Extract embedding from response.
model_response = json.loads(response["body"].read())
embedding = model_response["embedding"]

# Query vector index with a metadata filter.
response = s3vectors.query_vectors(
    vectorBucketName="media-embeddings",
    indexName="movies",
    queryVector={"float32": embedding},
    topK=3,
    filter={"genre": "scifi"},
    returnDistance=True,
    returnMetadata=True
)

print(json.dumps(response["vectors"], indent=2))

We also added a metadata filter on the genre field. The final result looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[
  {
    "distance": 0.7918924689292908,
    "key": "Star Wars",
    "metadata": {
      "source_text": "Star Wars: A farm boy joins rebels to fight an evil empire in space",
      "genre": "scifi"
    }
  },
  {
    "distance": 0.8599859476089478,
    "key": "Jurassic Park",
    "metadata": {
      "genre": "scifi",
      "source_text": "Jurassic Park: Scientists create dinosaurs in a theme park that goes wrong"
    }
  }
]

Great, right? With just a few API calls, we get semantic search plus metadata filtering — a practical building block for cost-sensitive RAG systems.

Summary

S3 Vector combines S3’s low-cost storage with Serverless vector search. It is a strong option for building cost-sensitive RAG systems — especially when you have large volumes of vector data, moderate query traffic, and relaxed latency requirements.

As I mentioned in my previous RAG articles, choosing the right vector store is always a trade-off. If your workload does not need millisecond-level retrieval on every query, S3 Vector is worth a closer look.

Background

If you have been reading my articles for a while, you probably know that I have focused mainly on Azure and AWS cloud AI services and applications. In this post, I want to step outside that comfort zone and introduce a different direction: edge AI.

In this article, I will share what I learned about ONNX (Open Neural Network Exchange) — an open standard and file format that helps AI models run across platforms. We will look at the problem it solves, the convenience it brings to AI/ML applications, and how this capability extends in the GenAI era to make local small-model deployment and inference much easier.

If those questions interest you, let’s get started.

ONNX: Cross-Platform AI Models

What Problem Does ONNX Solve?

What problem is ONNX trying to solve? Let me explain with a story from my own experience.

At a previous employer, we built a real-time computing system as a cloud service on the .NET stack. Due to business requirements, we needed to explore deploying machine learning models on that system and serving online inference. Before that, the system had no ML requirements or capabilities at all — it was a fairly ordinary .NET backend microservices stack.

To add ML inference, I researched several approaches:

Dedicated model inference services: Platforms like Azure Machine Learning or AWS SageMaker can host models for online inference, and our system would communicate with them over the network. This approach requires significant infrastructure investment, and network calls also add latency.

Embedded ML frameworks: We could integrate frameworks like PyTorch or TensorFlow directly into our system, embed models in-process, avoid extra infrastructure and operations, and skip network calls at inference time — all good for performance. But this approach locks you into a specific framework and reduces flexibility.

As you can see, each approach has trade-offs. Wouldn’t it be great if one solution combined the strengths of both while avoiding their weaknesses? That is exactly what ONNX does.

ONNX

ONNX (Open Neural Network Exchange) is an open standard that defines a common format for machine learning models, enabling cross-platform deployment. You can think of it as the AI equivalent of POSIX (Portable Operating System Interface) — a standard that lets different UNIX/Linux kernels interoperate.

Today, ONNX is supported by major ML frameworks and vendors.

ONNX & ONNX Runtime

With ONNX, the ML training and deployment workflow naturally splits into two phases:

Training phase: ONNX defines the model standard, and mainstream ML frameworks provide tools to convert models into ONNX format. In the example below, the scikit-learn framework uses the skl2onnx library to perform the conversion.

1
2
3
4
5
6

# Convert into ONNX format.
from skl2onnx import to_onnx

onx = to_onnx(clr, X[:1])
with open("rf_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

Inference phase: Another key component is ONNX Runtime — the runtime engine that executes models. ONNX Runtime is a native C++ library with binding APIs for different stacks, such as C# and Python.

In addition, ONNX Runtime provides different libraries for different hardware — CPU, GPU, and more. In this article, we will default to CPU. In a future post, I will dig into how to improve efficiency with GPU.

The most basic inference flow looks like this. This example uses .NET and C#, based on the Microsoft.ML.OnnxRuntime NuGet package.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

// 1. Create an inference session
session = new InferenceSession(onnxModelPath)

// 2. Build tensor from features
var inputTensor = new DenseTensor<float>(data, shape);

// 3. Wrap for ONNX Runtime (legacy API)
var featuers_input = NamedOnnxValue.CreateFromTensor<float>("feature_input", inputTensor);

// 4. Run model
session.Run(features_input);

Using ONNX for traditional ML models is not the main focus of this article. Next, let’s look at what opportunities ONNX opens up in the GenAI era.

ONNX Runtime GenAI: Evolution in the LLM Era

In the GenAI era, ONNX Runtime has a dedicated extension: ONNX Runtime GenAI. It follows the same philosophy as ONNX Runtime — letting applications embed and load models directly.

In theory, that means large models too. But frontier models today sit in the hundreds of billions or trillions of parameters. In practice, no application can realistically embed those locally.

We can, however, push the idea to the other extreme: small models!

Because they do not depend on cloud resources, small models fit many scenarios: limited network connectivity, strict data privacy and compliance, and lower accuracy requirements but higher latency sensitivity, among others.

So ONNX Runtime GenAI is a strong platform for edge AI built on small models!

In the demo below, let’s try it hands-on.

Demo: Local Small Models with ONNX

Phi-3-Mini-4K-Instruct ONNX Model

In this article, we use Microsoft’s Phi family small model: Phi-3-Mini-4K-Instruct.

Phi-3-Mini-4K-Instruct is an instruction-tuned variant built on top of Phi-3-Mini-4K — a model better aligned with how users interact with chat systems. It has 3.8B parameters and a 4K context window.

The model ships in variants for different compute targets: cuda, directml, and cpu_and_mobile.

Today we use the cpu_and_mobile variant. It is quantized, with weights compressed to int4, which makes it friendly for ordinary PCs, mobile devices, and edge hardware.

Building a Transformer Workflow with ONNX Runtime GenAI

When you call large models on cloud platforms like Azure Foundry or AWS Bedrock, you usually just specify the model name. Some platforms expose a few Transformer parameters — temperature, top_k, top_p, and so on — so you can shape model behavior at a basic level.

ONNX Runtime GenAI is a relatively low-level engine. You need to assemble the Transformer workflow yourself through its APIs.

On one hand, that is a real challenge. On the other hand, it is an excellent way to learn and practice Transformer fundamentals hands-on. In the sections below, let’s see how to build that workflow with ONNX Runtime GenAI.

The code examples use C# and .NET.

Creating the Tokenizer

The basic unit of information for large language models is the token, so the first step is to create a Tokenizer.

1
2
3
4
5
6

using Microsoft.ML.OnnxRuntimeGenAI;

var tokenizer = new Tokenizer(model);
var tokenizerStream = tokenizer.CreateStream();

Console.WriteLine("Tokenizer created");

Later, we will see how this Tokenizer converts the user’s prompt into tokens.

Creating the Model

Next, we create the model.

1
2
3
4
5
6

using Microsoft.ML.OnnxRuntimeGenAI;

var path = "pathtoonnxmodel"
var config = new Config(path);
var model = new Model(config);
Console.WriteLine("Model loaded");

Pass the model directory path to the engine. It automatically loads the ONNX Runtime GenAI configuration file: genai_config.json. For the Phi-3-Mini-4K-Instruct model we use here, the file looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

{
    "model": {
        "bos_token_id": 1,
        "context_length": 4096,
        "decoder": {
            "session_options": {
                "log_id": "onnxruntime-genai",
                "provider_options": []
            },
            "filename": "phi3-mini-4k-instruct-cpu-int4-rtn-block-32-acc-level-4.onnx",
            "head_size": 96,
            "hidden_size": 3072,
            "inputs": {
                "input_ids": "input_ids",
                "attention_mask": "attention_mask",
                "past_key_names": "past_key_values.%d.key",
                "past_value_names": "past_key_values.%d.value"
            },
            "outputs": {
                "logits": "logits",
                "present_key_names": "present.%d.key",
                "present_value_names": "present.%d.value"
            },
            "num_attention_heads": 32,
            "num_hidden_layers": 32,
            "num_key_value_heads": 32
        },
        "eos_token_id": [
            32000,
            32001,
            32007
        ],
        "pad_token_id": 32000,
        "type": "phi3",
        "vocab_size": 32064
    },
    "search": {
        "diversity_penalty": 0.0,
        "do_sample": false,
        "early_stopping": true,
        "length_penalty": 1.0,
        "max_length": 4096,
        "min_length": 0,
        "no_repeat_ngram_size": 0,
        "num_beams": 1,
        "num_return_sequences": 1,
        "past_present_share_buffer": true,
        "repetition_penalty": 1.0,
        "temperature": 1.0,
        "top_k": 1,
        "top_p": 1.0
    }
}

Beyond basic model metadata, the search section defines many parameters that affect Transformer inference behavior. We cannot cover all of them in this article, but keep in mind they are adjustable. Try changing them and observe the differences — that is one of the best ways to build intuition for how models behave. I will share more on this topic in a future post.

Configuring the Generator

Next we need a Generator. This is not a theoretical Transformer concept — it is a key engine component that orchestrates the entire workflow.

Transformer models use an autoregressive mechanism: generate one token at a time, then predict the next token based on the prompt and all tokens generated so far — forming a loop. The Generator controls that loop.

When you create the Generator, you can tune the parameters mentioned above to control model behavior:

1
2
3
4

using Microsoft.ML.OnnxRuntimeGenAI;

var generatorParams = new GeneratorParams(model);
var generator = new Generator(model, generatorParams);

Now let’s look at that loop.

Token Generation Loop

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

using Microsoft.ML.OnnxRuntimeGenAI;

// Get input tokens
string user_prompt = "hello";
string prompt = $@"[{{""role"":""system"",""content"":""{systemPrompt}""}},{{""role"":""user"",""content"":""{user_prompt}""}}]";
var sequences = tokenizer.Encode(prompt);

// append input tokens to generator
generator.AppendTokenSequences(sequences);

// Run generation loop
while (true)
{
    generator.GenerateNextToken();
    if (generator.IsDone())
        break;
}

// Get output tokens and decode to string
var outputSequence = generator.GetSequence(0);
var outputString = tokenizer.Decode(outputSequence);

// Display output
Console.WriteLine("Output:");
Console.WriteLine(outputString);

From the code above, the flow breaks down into these steps:

• Use the Tokenizer to convert the user prompt into a token sequence
• Pass the token sequence to the Generator
• The Generator produces the next token one at a time until completion
• Read the full sequence from the Generator
• Use the Tokenizer to decode the token sequence back into a string

This code is a clear, hands-on illustration of how Transformer inference works in practice. It is worth reading carefully more than once.

Build and run it, and you will see a small model answering your questions locally on ordinary CPU hardware. Great, right?