Docs/consumer / limits

Usage Limits & Best Practices

To keep production JSON feeds fast and fair, the PipeAgent gateway applies tiered rate limits and usage policies.

1. Rate limiting

The gateway uses Upstash-backed sliding windows (approximately per calendar minute), not a separate burst allowance. Limits below are maximum successful requests in that window.

Consumer reads (`GET /v1/feed/{id}`)

Enforced per user, per feed (your read API key identifies the user).

Tier	Applies when	Limit
Free	Feeds with no per-call metering (typically free marketplace feeds)	10 requests / minute
Paid	Metered feeds (each `GET` consumes credits) or one-time purchase feeds (you have completed the one-time buy; access is not metered per call)	100 requests / minute

Provider push (`POST /v1/provider/feed/{id}/push`)

Per feed (not per user): 20 requests / minute. See Provider API — for AI Agents.

Rate Limit Headers

Consumer GET /v1/feed/{id} responses include headers to help you manage throughput:

X-RateLimit-Limit: The maximum number of requests allowed in the window.

X-RateLimit-Remaining: The number of requests remaining in the current window.

X-RateLimit-Reset: The time at which the current rate limit window resets (UTC Epoch).

Handling 429 Errors

If you exceed your limit, the API will return a 429 Too Many Requests status code.

💡

TIP

Best Practice: Implement Exponential Backoff in your agent's retry logic. Use the retry_after_ms field in the JSON response to calculate your wait time.

2. Pricing & Credits

Paid feeds bill through a micro-credit model.

Standard Pricing: Most paid feeds cost $0.001 per 1,000 calls.

Credit Value: 1 Credit = $0.000001 USD.

Promotion Credits: Users with a valid promotion code may receive free credits to test paid feeds.

If your balance is insufficient for a paid feed, the API will return a 402 Payment Required error with a link to the billing dashboard.

3. Query Optimization

To save on latency and data processing costs, use the following optional parameters:

Pagination

If a feed returns an array, use limit and offset to keep payloads small.

limit: Maximum 100 items per request (default: 20 on some endpoints; values above 100 are capped at 100).

offset: Defaults to 0.

cursor: Opaque pagination cursor for collection and stream feeds.

Stream Time Range

For stream feeds, use native time-window filters on event_timestamp:

start_time: inclusive lower bound (event_timestamp >= start_time)

end_time: inclusive upper bound (event_timestamp <= end_time)

Both values must be valid ISO-8601 datetime strings.

bash

GET https://api.pipeagent.dev/v1/feed/{id}?limit=20&start_time=2026-03-24T00:00:00Z&end_time=2026-03-24T23:59:59Z

JSONPath Filtering

You can filter the JSON response at the edge using the jsonpath parameter. This prevents your AI agent from receiving irrelevant "noise" and saves LLM token costs.

bash

GET https://api.pipeagent.dev/v1/feed/{id}?jsonpath=$[*].{title: @.name, cost: @.price}

4. Provider push (collection, stream & singleton)

When using POST https://api.pipeagent.dev/v1/provider/feed/{id}/push, the gateway enforces raw HTTP body size (bytes on the wire) plus, for collection and stream feeds, per-row and row-count guardrails. Limits below are defaults; deployments may override them with environment variables.

Singleton feeds

Guardrail	Default	Environment variable
Max request body	4 MB	`PROVIDER_PUSH_SINGLETON_MAX_BODY_BYTES`

If the body is too large, use a collection feed, split the data, or trim fields.

Collection & stream feeds

Guardrail	Default	Environment variable
Max request body	2 MB	`PROVIDER_PUSH_MAX_REQUEST_BODY_BYTES`
Max JSON size per row	50 KB (UTF-8 bytes of `JSON.stringify(row)`)	`PROVIDER_PUSH_MAX_ROW_BYTES`
Max rows per request (abuse / overload protection)	500	`PROVIDER_PUSH_MAX_ROWS`

Normal integrations should not hit the row cap; split large backfills into multiple requests. The per-row and request body limits apply together: many small rows can still hit the body limit before the row cap.

Error shape (413)

Push responses may include a limit field: singleton_body, request_body, row_count, or row_payload, plus numeric hints such as max_bytes, max_rows, or row_index.

5. Error Codes Reference

Code	Meaning	Action
401	Unauthorized	Missing or malformed `x-api-key` header.
402	Payment Required	Paid feed requested, but balance is $0. Link a payment method.
403	Forbidden	API key is valid but does not have `read` permissions for this feed.
404	Not Found	The feed ID does not exist or has not been synced yet.
429	Too Many Requests	Rate limit exceeded. Wait for the window to reset.
500	Internal Error	Something went wrong on our end. Contact support if persistent.
413	Payload Too Large	Push body or a single row exceeds configured limits (see Provider push above).

6. Data Provenance

All consumer responses include an _origin object with usage_id and timestamp.

Transparency: You can use the usage_id to audit your billing logs in the dashboard.

License: All data is provided under the PipeAgent Dynamic Data License. Please respect attribution and redistribution constraints. See Data License.

---

*Need higher limits for an enterprise use case? Contact our sales team for a custom quota.*

Version 1.0.4 - Premium Infrastructure

Legal Disclaimer