Add the AI SDK with coroutine-safe execution and framework integrations
This adds hypervel/ai, porting Laravel's AI SDK to Hypervel with coroutine-safe execution, persistent conversations, and integrations with Scout, Sentry, and Telescope. Applications can use the familiar agent and provider APIs while running concurrent requests in long-lived workers.
The package includes text and structured output, streaming, tools and approvals, conversation history, images, speech and transcription, embeddings, reranking, classification, file stores, skills, testing fakes, and queued generation. It also includes runtime middleware and tools, provider failover, and AG-UI and Vercel AI SDK chat inputs and stream protocols. Provider support covers OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Cohere, DeepSeek, ElevenLabs, Groq, Jina, Mistral, Ollama, OpenAI-compatible endpoints, OpenRouter, TypeSafe, VoyageAI, and xAI, according to each provider's capabilities.
Execution and provider ownership
Request state belongs to an operation rather than a shared worker. Provider instances and their credentials survive failover, deferred execution, and file-store operations without being replaced by a fresh lookup of the provider name. Repeated provider/model entries remain separate failover attempts.
Applications can resolve provider configuration for each operation, scope embedding caches, partition stored conversations, and capture application context for deferred work. These hooks support applications with different credentials and data boundaries for different customers without adding a tenant framework to the package. Runtime middleware can be attached to an agent instance without modifying its class; each invocation captures the instance's middleware.
Provider traffic uses Hypervel's HTTP client. Streaming stays incremental, including tool follow-up requests, and releases response bodies on completion, failure, cancellation, or abandonment. Bedrock keeps credential reuse and refresh coordination with the provider while creating SDK clients for individual operations. AWS signing and credential-chain behavior remain with the AWS SDK.
Creating fresh Bedrock clients has a measured cost. With inherited AWS shared configuration, client CPU per buffered step was 3.98 ms versus 0.60 ms in a shared-client experiment; streamed steps were 4.99 ms versus 1.47 ms. Without that configuration, reuse saved 0.8–0.9 ms. Fresh clients keep credentials and concurrent request promises owned by their operations without adding shared-client coordination.
Remote attachments use the HTTP client's shared destination checks and a configurable decoded-download limit, defaulting to 32 MiB. Embedded URL credentials are rejected. Explicitly trusted hosts can be listed in ai.remote_files.allowed_hosts. If an outgoing proxy resolves destination hostnames, that proxy must filter the addresses it connects to.
Provider parsing also fixes concrete failures: incomplete streams no longer look like successful responses, terminal errors reach callers, multiple text parts are retained, zero-valued content is preserved, and top-level tool arguments remain JSON objects. Other corrections cover reasoning replay, Gemini file-search filters, and image attachment capability checks.
Streaming CPU and memory
Stream response aggregation runs in one pass. Response callbacks no longer form reference cycles that retain completed streams until cyclic garbage collection. Replay retains the original events.
SSE and protocol responses enable disconnect cancellation to stop provider work on supported whole-connection closes. An individual HTTP/2 stream reset is detected only on a failed write, completion, or timeout with the current runtime.
The following measurements combine those AI changes with the receive-buffer copying fix already merged into 0.4 in #669. They show the combined effect; they aren't an isolated measurement of this PR against current 0.4.
| Local workload / metric | Before | After both changes |
|---|---|---|
| Small paced response, client CPU/request | 6.88 ms | 5.66 ms |
| 1 MiB generated text, client CPU/request | 1,260 ms | 687 ms |
| Same large workload, end-of-run PHP heap | 49.82 MiB | 9.17 MiB |
| Concurrent account churn, client CPU/request | 9.63–9.66 ms | 5.56–5.60 ms |
| Same churn, completed requests/second | 103.5–103.8 | 178.4–179.8 |
| Same churn, cyclic GC runs per sample | 149–154 | 0 |
Heap is measured at the end of a run, not per request, and excludes native allocations. The large response includes the provider protocol's terminal complete-response payload.
Conversation storage and tool approvals
A conversation turn is stored atomically. The store writes the conversation timestamp once while still calling the public message-write methods, so subclass overrides continue to work. Parent-first row locking gives concurrent writes a consistent lock order. Conversation partition checks apply to reads, writes, and restoration; deleting a conversation cascades to its messages.
Stored approval resumes use an exclusive claim. Each completed tool result is recorded before proceeding, so a later failure doesn't discard earlier results. This costs additional commits on approval resumes: recording five results took 28.4 ms on PostgreSQL and 63.8 ms on MySQL in the storage fixture below. It does not promise exactly-once external side effects: a process can fail after an external action but before saving its result. An uncertain execution isn't silently retried.
Paused-message scans are paged, and replay cleanup reads flagged message IDs instead of decoding every paused message. The migration uses UUID identifiers, configurable participant key types, long-text payloads, approval claim and replay fields, and indexes matched to the store's queries.
| Serial turn persistence | Separate public message writes | Atomic turn with one conversation update |
|---|---|---|
| PostgreSQL wall time | 12.60 ms | 5.89 ms |
| PostgreSQL client CPU | 1.88 ms | 1.45 ms |
| MySQL wall time | 24.30 ms | 14.13 ms |
| MySQL client CPU | 1.96 ms | 1.55 ms |
This compares two Hypervel paths, not Hypervel against Laravel. The database timings were collected on a nearly full Docker volume and shouldn't be treated as general database latency predictions.
Caching and optional concurrency
Fixed skill directories are cached for the worker lifetime; other skill sources stay dynamic. Deployments reload services when cached skill files change. Embedding cache identities are computed once for a batch and distinguish provider configuration and application scope.
Tool execution and conversation-title generation can run concurrently through #[ConcurrentTools(max: …)] and ai.conversations.concurrent_title. Tools have bounded concurrency and retain result order. Title generation overlaps the response and is joined or cancelled with its owning operation. Enabling concurrent titles can incur a title request even when an attempt fails, including a separate request for each failover attempt. Child operations carry application context and the scoped database connection name without sharing the parent's transaction.
| Controlled workload | Before | After |
|---|---|---|
| Discover 20 skills: uncached → cached | 4,150 µs | 24.8 µs |
| Discover 100 skills: uncached → cached | 20,348 µs | 103 µs |
| Four 20 ms tools: serial → concurrent | 81.7 ms | 21.2 ms |
| Response and title, including SQLite persistence: serial → concurrent | 44.6 ms | 23.3 ms |
These are local filesystem and controlled-wait measurements, not estimates of model response times. Tool and title concurrency remain opt-in.
Broadcast batching
Broadcasts deliver immediately by default; callers can still choose queued delivery with now: false. Adjacent text and reasoning deltas are batched by default with a 50 ms window and a 4,096-byte limit on JSON-encoded delta content. The dedicated ai.broadcasting.batching configuration can change these limits or disable batching.
One bounded sender handles a stream. Structural events keep their order, original events remain available for replay, and SSE delivery is unchanged. During applicable transactions or event deferral, and outside coroutines, the caller sends events individually without batching. The window bounds deliberate batching delay, not total delivery time under backpressure.
The experiment below used sixteen streams producing 576 original events against a local Pusher-compatible HTTPS receiver. Connection retention was already enabled in both variants. The measured batching helper informed the production implementation; these are experiment results, not a new measurement of the final implementation.
| Delivery / receiver delay | Sends or jobs, off → 50 ms | Client CPU, off → 50 ms | Event acceptance p95, off → 50 ms |
|---|---|---|---|
| Immediate / 5 ms | 576 → 240 | 311.82 → 149.27 ms | 22.51 → 58.41 ms |
| Immediate / 50 ms | 576 → 224 | 325.26 → 145.16 ms | 1,263.91 → 156.63 ms |
| Queued / 5 ms | 576 → 240 | 945.65 → 411.44 ms | 495.22 → 76.11 ms |
| Queued / 50 ms | 576 → 240 | 1,028.39 → 428.73 ms | 1,264.32 → 171.97 ms |
Batching cuts sends/jobs by roughly 58–61% and client CPU by 52–58% here. The tradeoff is visible in the fast immediate case: individual deltas arrive later, although the last event is accepted at about 663 ms either way. In the slower immediate case, last-event acceptance falls from about 1,989 ms to 818 ms. Queued producers and workers shared a process in this fixture; these aren't forecasts for independently deployed workers.
Scout embeddings and vector search
Scout can generate embeddings through the SDK for Meilisearch, Typesense, and Turbopuffer, alongside their existing native and precomputed embedding modes. The database engine adds PostgreSQL/pgvector semantic and hybrid search, embedding updates, and pagination. Hybrid search loads up to 1,000 text and 1,000 semantic candidates and paginates the first 1,000 fused results. It preserves query customizations and model hydration; applications can select only the fields they need to reduce candidate memory use.
Database-vector imports limit child concurrency using the imported models' actual connection pool, reserving a slot for the parent chunk reader. A one-connection pool executes inline. Before this limit, concurrency 50 with a 10-connection pool failed a 5,000-row import after 4,500 updates. The fix has regression coverage but wasn't benchmarked for throughput. All concurrent Scout imports now preserve DB::usingConnection() selection.
Embedding writes retain per-row commits: a 100-row transaction experiment reduced generated import time from 22.65 to 18.65 seconds but increased overlapping writer p99 from 170 to 1,455 ms. Those figures share the database-storage caveat above. Models using the Searchable trait gain searchableEmbeddingColumn() automatically; direct implementers of SearchableInterface must add it.
Sentry tracing and Telescope redaction
Sentry records invocation, model-step, tool, embedding, and classification spans. Model steps use already-resolved instructions and tools, and HTTP spans sit under the relevant step. Streaming spans follow consumption and close on success, failure, cancellation, or abandonment. Nested agents restore the outer agent's current span and conversation, and each failover attempt keeps its own spans.
The gen_ai* tracing switches are enabled by default when the AI package is installed, subject to Sentry's tracing and sampling settings. With the default data_collection: null, prompts, responses, tool payloads, and embedding/classification content are omitted unless send_default_pii is enabled. An explicit data_collection configuration instead controls them through gen_ai.inputs and gen_ai.outputs, which default to true if omitted.
The integration avoids work when tracing or individual span families are disabled. Enabled listeners check tracing state before formatting payloads, and AI lifecycle event construction is guarded by hasListeners(). History formatting is lazy and bounded, attachment contents aren't loaded for telemetry, and bounded JSON remains valid rather than being cut mid-document. Bounded maps cap abandoned records; weak references keep those records from retaining their coroutine tracing scopes. These are architectural resource controls; no Sentry throughput improvement is claimed from the AI benchmarks.
The AI events are also usable independently of Sentry: StartingStep includes resolved instructions and tools, streamed tools dispatch ToolFailed, and new AgentStreamAbandoned, ClassificationFailed, and EmbeddingsFailed events expose terminal paths to listeners.
Telescope masks additional AI, AWS, OAuth, and search-provider credentials in headers and payloads. Query credentials are also hidden in recorded URLs, redirects, relevant headers, and session navigation URLs without rewriting unrelated query bytes.
Supporting framework fixes
- HTTP streaming reuse selects idle connections by route as well as proxy tunnel identity, avoiding connection churn when alternating origins. Destination checks handle DNS64-translated addresses and scheme-specific proxy settings, and accept hostname labels containing underscores while retaining address checks.
- Permission partitions are stamped on
saveOrIgnore()as well as ordinary inserts. Post-insert listeners can update a newly created model without being mistaken for a partition change. - Queue failure hooks in listeners, mail, notifications, and closures accept a missing exception, allowing manual failures to reach their callbacks.
- JSON Schema deserialization preserves nested nullable
anyOfcompositions. Databasetake()andskip()accept null consistently withlimit()andoffset(). - Testbench starts Workbench before database setup, links dependencies into isolated runtime copies, accepts empty YAML collections, and writes portable boolean environment values. Serial test cleanup doesn't require ParaTest.
The package includes an isolated dogfood/ai browser demo for contributors, package discovery/configuration wiring, and upstream-derived tests and fixtures. PostgreSQL CI services include pgvector for real vector-search coverage.
At the root package level, symfony/yaml becomes a runtime dependency, and a Composer conflict excludes aws/aws-sdk-php versions below 3.369.1 for all hypervel/components installations.
The AI SDK documentation is ported alongside updates to the HTTP client, Scout, Sentry, Telescope, Permission, JSON Schema, and Laravel porting pages.
Compatibility and validation
The usual public generation APIs follow Laravel. Intentional differences are documented in the package README and porting guide, including provider/model pair returns, protected extension points, remote attachments, skill caching, broadcasting, and custom approval-store requirements. Existing Laravel conversation tables need the documented schema migration; they aren't interchangeable with the new schema.
Formatting, static analysis, targeted package tests, Testbench, and the demo tests passed. Coverage includes real loopback streaming, cancellation and cleanup, database locking and partition isolation, pgvector, queued execution, and Sentry span ownership. The final full-suite run stalled in a worker on Swoole 6.3.0RC1, which has a tracked shutdown hang; that run isn't reported as a completed pass. Live paid-provider tests and optional MCP integration weren't exercised locally.
Benchmarks use local mock providers and services, with no paid AI accounts. Repeated streaming and broadcast results are medians from alternating fresh-process comparisons on PHP 8.4.25/Swoole 6.3.0RC1 with CLI OPcache enabled and JIT disabled. Reproduction commands, full results, and measurement limits are in tests/Benchmarks/Ai. Client CPU excludes separate service processes. One MySQL process that hit the normal pool checkout deadline was excluded from the storage medians.
Known AWS SDK cancellation and event-stream decoder defects have focused skipped regressions with upstream issue links rather than local workarounds. Bedrock streams can fail with a checksum error or lose the final usage/error event; see the known issues. The upstream limitation around nested JSON object/list shapes in tool arguments is also documented. The included MCP adapters require the Hypervel MCP package, which isn't available yet.
[!NOTE]
Add hypervel/ai SDK with providers, agents, tools, and framework integrations
- Introduces a full AI SDK package (src/ai) covering agent prompting and streaming, structured output, tool approvals, conversation persistence with partitioning, and gateways for OpenAI, Anthropic, Gemini, Bedrock, Groq, Mistral, Cohere, DeepSeek, Ollama, xAI, VoyageAI, Jina, ElevenLabs, OpenRouter, Azure, and OpenAI-compatible endpoints.
- Adds a test double layer (
fake/assert APIs) for agents, audio, embeddings, files, images, reranking, stores, transcription, and classification, plus a Vercel/AG-UI chat input adapter and streaming SSE/AG-UI response protocols.
- Integrates the SDK elsewhere: Scout engines (Meilisearch, Typesense, Turbopuffer, Postgres HNSW) generate embeddings through the AI SDK for semantic/hybrid search; Sentry gains gen_ai tracing spans with a data-collection policy; Telescope redacts credential query parameters in recorded URIs, sessions, and redirect responses.
- Fixes prompt rendering (src/prompts) to route frames through a decoration-aware writer, measure visible widths after formatter markup, and escape literal formatter tags so rendered output is not re-interpreted.
- Includes an AI Workbench dogfood app (dogfood/ai) with a React chat UI, browser chat and AG-UI routes, and demo tools.
- Risk: adds
hypervel/ai and symfony/yaml as root dependencies in composer.json and conflicts with AWS SDK >= 3.369.1; Env::encode and parse_environment_variables now emit plain true/false instead of parenthesized strings, affecting anything parsing testbench env output; queued failed() handlers in CallQueuedListener, SendQueuedMailable, SendQueuedNotifications, and CallQueuedClosure now accept a nullable Throwable; Testbench Workbench setup now runs before database traits via setUpTheTestEnvironmentTraitToBeIgnored.
Macroscope summarized 1544117.