Agents on AWS SageMaker AI with Strands Agents

Alvaro Bartolome

alvarobartt

Follow

In this post I'll walk through deploying an open model from the Hugging Face Hub with vLLM on AWS SageMaker AI, then using that endpoint to power a Strands agent. Along the way I'll also show how to wire in tools, connect a Hugging Face MCP server, and inspect the agent's reasoning and tool calls from a small Gradio app.

SageMaker AI gives you two pretty different ways to run open models from the Hugging Face Hub. If you want the fastest path, SageMaker JumpStart is the more convenient route. If you care about the serving image, runtime flags, parser settings, or generally not being boxed into the happy path, custom deployment is the more interesting option. That's the route I'll use here. If you want the more guided experience, SageMaker JumpStart is the place to start.

Prerequisites and AWS Setup

You'll need AWS access with SageMaker AI and IAM permissions, enough quota for the GPU instance type you want, Python 3.10+, and the AWS CLI installed. On the Python side, I'll use boto3 and the SageMaker SDK.

• An AWS account with access to SageMaker AI and IAM
• Enough quota for the GPU instance type you plan to use
• The AWS CLI installed and authenticated

If you haven't authenticated with AWS on your machine yet, the simplest option is usually AWS IAM Identity Center with:

aws configure sso

That will walk you through selecting your start URL, region, account, and role, and it will save a profile locally for reuse. If you already use long-lived access keys, that works too, for example via aws configure or environment variables such as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION.

Then install the Python dependencies:

pip install sagemaker boto3 --upgrade --quiet

Before creating any SageMaker AI resources, you'll also want an execution role ARN that SageMaker can assume on your behalf. A quick way to create one is to first check whether the role already exists:

aws iam get-role --role-name hf-aws-role

If that returns a NoSuchEntity error, create it and attach the managed SageMaker policy:

aws iam create-role \
  --role-name hf-aws-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sagemaker.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

Then attach the managed SageMaker policy so the service can assume the role and create the model, endpoint configuration, and endpoint resources used in this post.

aws iam attach-role-policy \
  --role-name hf-aws-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess

Deploy Open Models on AWS SageMaker AI

boto3 will pick your AWS crendentials automatically from the default credential chain, so there is no extra authentication code you need for the deployment itself. In practice, the main value of the following snippet is just to fetch the role_arn that SageMaker AI needs when creating the model resource:

import boto3

session = boto3.Session()
iam = session.client("iam")

role = iam.get_role(RoleName="hf-aws-role")
role_arn = role["Role"]["Arn"]

In the snippet below, the vLLM container is configured directly through environment variables with the SM_VLLM_* prefix, which are mapped on runtime to their counterpart arguments in vLLM e.g., SM_VLLM_MODEL is mapped to --model on runtime.

from sagemaker.core.resources import Endpoint, EndpointConfig, Model
from sagemaker.core.shapes import ContainerDefinition, ProductionVariant

model = Model.create(
    model_name="vllm-model",
    primary_container=ContainerDefinition(
        image="763104351884.dkr.ecr.us-east-1.amazonaws.com/vllm:server-sagemaker-cuda-v1",
        environment={
            "SM_VLLM_MODEL": "Qwen/Qwen3.6-27B",
            "SM_VLLM_TENSOR_PARALLEL_SIZE": 4,
            "SM_VLLM_MM_ENCODER_TP_MODE": "data",
            "SM_VLLM_MAX_MODEL_LEN": "262144",
            "SM_VLLM_TOOL_CALL_PARSER": "qwen3_coder",
            "SM_VLLM_ENABLE_AUTO_TOOL_CHOICE": "true",
            "SM_VLLM_REASONING_PARSER": "qwen3",
            "SM_VLLM_ENABLE_LOG_REQUESTS": "true",
            "VLLM_LOGGING_LEVEL": "INFO",
        },
    ),
    execution_role_arn=role_arn,
)

endpoint_config = EndpointConfig.create(
    endpoint_config_name="vllm-config",
    production_variants=[
        ProductionVariant(
            variant_name="default",
            model_name="vllm-model",
            instance_type="ml.g6e.12xlarge",
            initial_instance_count=1,
            inference_ami_version="al2-ami-sagemaker-inference-gpu-3-1",
        ),
    ],
)

endpoint = Endpoint.create(endpoint_name="vllm-endpoint", endpoint_config_name="vllm-config")
endpoint.wait_for_status("InService")

And voilà!

Send Request to the AWS SageMaker Endpoint

If you're not familiar with AWS SageMaker AI, you might expect the endpoint to be directly OpenAI-compatible and send requests to either /v1/chat/completions or /v1/responses, but the SageMaker Runtime API works a bit differently as everything goes through /invocations.

E.g. From Python you would call invoke_endpoint, which sends a JSON payload in the request body and uses CustomAttributes to route the request to the underlying Chat Completions API.

import json

import boto3

session = boto3.Session()
client = session.client("sagemaker-runtime")

response = client.invoke_endpoint(
    EndpointName="vllm-endpoint",
    ContentType="application/json",
    Body=json.dumps({
        "model": "Qwen/Qwen3.6-27B",
        "messages": [{"role": "user", "content": "What is Deep Learning?"}],
        "max_tokens": 256,
    }),
    CustomAttributes="route=v1/chat/completions",
)
print(json.loads(response["Body"].read()))

Alternatively, if you want to stream the chat completions, just adding stream: true to the body won't be enough because invoke_endpoint is not expecting a stream of chunks back. To actually stream tokens, you need to set stream: true and call invoke_endpoint_with_response_stream instead.

import json

import boto3

session = boto3.Session()
client = session.client("sagemaker-runtime")

response_stream = client.invoke_endpoint_with_response_stream(
    EndpointName="vllm-endpoint",
    ContentType="application/json",
    Body=json.dumps({
        "model": "Qwen/Qwen3.6-27B",
        "messages": [{"role": "user", "content": "What is Deep Learning?"}],
        "max_tokens": 256,
        "stream": True,
    })
)

for event in response_stream["Body"]:
    if payload_part := event.get("PayloadPart"):
        chunk = payload_part["Bytes"].decode()
        if chunk.startswith("data: "):
            chunk = chunk[6:]

        try:
            data = json.loads(chunk)
        except json.JSONDecodeError:
            continue

        ...

More information can be found in the Amazon SageMaker Runtime Documentation.

from openai import OpenAI
from sagemaker.core.token_generator import generate_token

openai_client = OpenAI(
    base_url=f"https://runtime.sagemaker.us-east-1.amazonaws.com/endpoints/{endpoint.endpoint_name}/openai/v1",
    api_key=generate_token(region="us-east-1"),
)

response = openai_client.chat.completions.create(
    model="Qwen/Qwen3.6-27B",
    messages=[
        {"role": "user", "content": "What is Deep Learning?"},
    ],
    stream=False,
)
print(response.choices[0].message.content)

Create Agents With Strands Agents

Now that the endpoint is up, the next step is making it useful inside an agent runtime. I like Strands Agents here because it already has a SageMaker AI model integration, so you don't have to build a custom transport layer before you even get to tools, memory, or tracing.

At the time I put this together, I also ended up using my own fork of the Strands Python SDK and opened this PR because reasoning parsing changed in vLLM v0.16+, so the reasoning traces with the current container, vLLM v0.20.1, where not parsed.

pip install "strands-agents[sagemaker] @ git+https://github.com/alvarobartt/sdk-python" --upgrade
pip install strands-agents-tools --upgrade

And, creating an agent is as easy as instantiating the SageMakerAIModel and creating the Agent with as much custom tools, MCP servers and/or integrations as you want.

from strands import Agent
from strands.models.sagemaker import SageMakerAIModel
from strands_tools import http_request, calculator

model_sagemaker = SageMakerAIModel(
    endpoint_config={"endpoint_name": "vllm-endpoint", "region_name": "us-east-1"},
    payload_config={
        "max_tokens": 81920,
        "temperature": 1.0,
        "top_p": 0.95,
        "stream": True,
        "additional_args": {
            "presence_penalty": 1.5,
            "top_k": 20,
            "chat_template_kwargs": {"enable_thinking": True},
        },
    },
)

agent = Agent(model=model_sagemaker, tools=[http_request, calculator], callback_handler=None)

There are two small details worth calling out in that snippet:

1. I set callback_handler=None because Strands uses a PrintCallback by default, which is handy when you're debugging in a terminal but gets noisy fast if you want to control the UI or capture events yourself.

2. chat_template_kwargs={"enable_thinking": True} is there to ensure that Qwen/Qwen3.6-27B operates in a thinking mode by default (it's already the default but it's there for consistency as other models as e.g., Qwen/Qwen3.5-0.8B come with a default non-thinking mode).

Define and Add Custom Tools

For tools, you can either use batteries-included ones from strands-agents-tools, like http_request and calculator, or define your own Python functions and register them as Strands tools. In practice I usually start with the built-ins to validate the model and endpoint setup, then swap in application-specific tools as needed.

Strands lets you easily define tools via the @tools decorator as it later derives the JSON schema from the implementation (including types, docstrings, and such).

from strands import Agent, tool

@tool
def weather_api(city: str, unit: str = "celsius") -> str:
    """Get the current weather for a city.

    Args:
        city: City to look up.
        unit: Temperature unit to return.
    """
    return f"The weather in {city} is 18 degrees {unit}."

@tool
def search_internal_docs(query: str) -> str:
    """Search an internal knowledge base.

    Args:
        query: Search query to run.
    """
    return f"Top result for '{query}': deployment runbook v2."

agent = Agent(
    model=model_sagemaker,
    tools=[weather_api, search_internal_docs],
    callback_handler=None,
)

For more on how the decorator turns functions into tools, see the official Strands tools overview and the @tool API reference.

Connect MCP Servers

pip install huggingface_hub mcp --upgrade

Strands can connect to MCP servers directly and expose their tools to the agent through MCPClient. E.g., The Hugging Face MCP server, because it can give your agent access to the Hugging Face Hub services, features and tools.

from huggingface_hub import get_token
from mcp.client.streamable_http import streamablehttp_client
from strands import Agent
from strands.tools.mcp import MCPClient
from strands_tools import calculator

mcp_client = MCPClient(
    lambda: streamablehttp_client(
        "https://huggingface.co/mcp",
        headers={"Authorization": f"Bearer {get_token()}"},
    )
)

agent = Agent(
    model=model_sagemaker,
    tools=[mcp_client],
    callback_handler=None,
)

response = agent(
    "Find a small Qwen model with tool-calling support and include the Hub links."
)
print(response)

For more information check the Strands Agents Model Context Protocol (MCP) Tools and Hugging Face MCP documentation.

Bonus: Create a Gradio + Strands Agents Application

If you want something a bit nicer than a terminal loop, a tiny Gradio app is enough. This version just consumes the Strands event stream and renders reasoning blocks, tool-use blocks, tool results, and final assistant text. It assumes the agent object from the previous section is already available.

import logging

import gradio as gr

logging.basicConfig(level=logging.WARNING)
logging.getLogger("httpx").setLevel(logging.WARNING)

async def run(prompt: str, messages: list | None):
    if messages is None:
        messages = []

    current_block_type = None
    current_tool_name = None
    content = ""

    async for event in agent.stream_async(prompt):
        if "event" in event:
            raw = event["event"]

            if "contentBlockStart" in raw:
                content = ""
                start = raw["contentBlockStart"].get("start", {})
                tool_info = start.get("toolUse")

                if tool_info:
                    if content.strip():
                        messages.append(gr.ChatMessage(role="assistant", content=content))
                    current_block_type = "tool"
                    current_tool_name = tool_info.get("name", "tool")
                    messages.append(gr.ChatMessage(
                        role="assistant",
                        content="",
                        metadata={"title": f"Using tool `{current_tool_name}`..."}
                    ))
                else:
                    current_block_type = "reasoning"
                    messages.append(gr.ChatMessage(
                        role="assistant",
                        content="",
                        metadata={"title": "Thinking..."}
                    ))

            elif "contentBlockDelta" in raw:
                delta = raw["contentBlockDelta"].get("delta", {})

                if current_block_type == "tool":
                    content += delta.get("toolUse", {}).get("input", "")
                    messages[-1] = gr.ChatMessage(
                        role="assistant",
                        content=content,
                        metadata={"title": f"Using tool `{current_tool_name}`..."}
                    )
                elif current_block_type == "reasoning":
                    reasoning = delta.get("reasoningContent", {})
                    text = reasoning.get("text", "") if isinstance(reasoning, dict) else delta.get("text", "")
                    content += text
                    messages[-1] = gr.ChatMessage(
                        role="assistant",
                        content=content,
                        metadata={"title": "Thinking..."}
                    )
                else:
                    content += delta.get("text", "")
                    if messages and messages[-1].metadata is None:
                        messages[-1] = gr.ChatMessage(role="assistant", content=content)
                    else:
                        messages.append(gr.ChatMessage(role="assistant", content=content))

            elif "contentBlockStop" in raw:
                if messages and not messages[-1].content.strip():
                    messages.pop()
                current_block_type = None
                current_tool_name = None
                content = ""

        elif "message" in event:
            msg = event["message"]
            if msg.get("role") == "user":
                for block in msg.get("content", []):
                    if "toolResult" not in block:
                        continue
                    result = block["toolResult"]
                    result_text = "\n".join(
                        item.get("text", "") for item in result.get("content", []) if "text" in item
                    ).strip()
                    if result_text:
                        status = result.get("status", "success").upper()
                        messages.append(gr.ChatMessage(
                            role="assistant",
                            content=result_text,
                            metadata={"title": f"Tool result [{status}]"}
                        ))

            elif msg.get("role") == "assistant":
                for block in msg.get("content", []):
                    text = block.get("text", "").strip()
                    if text and (not messages or text != messages[-1].content.strip()):
                        messages.append(gr.ChatMessage(role="assistant", content=text))

        yield messages


demo = gr.ChatInterface(fn=run)
demo.launch()

Clean Up the Resources

Finally, to avoid inadvertent charges, delete your endpoint and associated resources when you're done. SageMaker AI endpoints incur costs while they are InService, even if they are not receiving any traffic.

import boto3
session = boto3.Session()
client = session.client("sagemaker", region_name="us-east-1")

client.delete_endpoint(EndpointName="vllm-endpoint")
client.delete_endpoint_config(EndpointConfigName="vllm-config")
client.delete_model(ModelName="vllm-model"))

Conclusion

The nice thing about this setup is that it gives you a fairly direct path from an open model on the Hugging Face Hub to a working agent on managed infrastructure, without having to give up too much control over the runtime. You can pick the model, tune the vLLM serving configuration, expose tool-calling and reasoning features, and still end up with an endpoint that plugs into an agent framework cleanly.

Just as importantly, it keeps your options open. If you want a more guided experience, SageMaker JumpStart is there. If you want lower-level control, custom deployments are there. And if you want to wrap the endpoint in something agentic, Strands gives you a pretty practical starting point. In other words, this is a good way to experiment without prematurely locking yourself into a single deployment style or model-serving strategy.