---
title: Around
description: Search around a specific timestamp to fetch nearby log records using GET /_around with forward and backward context in a 5-minute window.
---

# Search around

Endpoint: `GET /api/{organization}/{stream}/_around?key={timestamp}&size=10`

## Request

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| stream     | string    | -             | stream name |
| key        | int64     | 0             | the `_timestamp` of the record what you want to search around |
| size       | int64     | 0             | how many records do you want to response around the record, we will search the record forward & backward 5 minutes |

## Response

```json
{
	"took": 155,
	"hits": [
		{
			"_p": "F",
			"_timestamp": 1674213225158000,
			"kubernetes": {
				"annotations": {
					"kubernetes": {
						"io/psp": "eks.privileged"
					}
				},
				"container_hash": "dkr.ecr.us-west-2.amazonaws.com/ziox@sha256:3dbbb0dc1eab2d5a3b3e4a75fd87d194e8095c92d7b2b62e7cdbd07020f54589",
				"container_image": "dkr.ecr.us-west-2.amazonaws.com/ziox:v0.0.3",
				"container_name": "ziox",
				"docker_id": "eb0983bdb9ff9360d227e6a0b268fe3b24a0868c2c2d725a1516c11e88bf5789",
				"host": "ip.us-east-2.compute.internal",
				"labels": {
					"app": "ziox",
					"controller-revision-hash": "ziox-ingester-579b7767cf",
					"name": "ziox-ingester",
					"role": "ingester",
					"statefulset": {
						"kubernetes": {
							"io/pod-name": "ziox-ingester-0"
						}
					}
				},
				"namespace_name": "ziox",
				"pod_id": "35a0421f-9203-4d73-9663-9ff0ce26d409",
				"pod_name": "ziox-ingester-0"
			},
			"log": "[2023-01-20T11:13:45Z INFO  actix_web::middleware::logger] 10.2.80.192 \"POST /api/demo/_bulk HTTP/1.1\" 200 68 \"-\" \"go-resty/2.7.0 (https://github.com/go-resty/resty)\" 0.001074",
			"stream": "stderr"
		}
	],
	"total": 10,
	"from": 0,
	"size": 0,
	"scan_size": 28943
}
```

Response description:

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| took       | int64     | -             | unit: milliseconds, query execute time |
| from       | int64     | 0             | value from `query.from` |
| size       | int64     | 0             | value from `query.size` |
| scan_size  | int64     | 0             | unit: MB, it response the data size scale when execute the query. |
| hits       | array     | -             | records for query, each record is a log row what you ingested. |

---
title: Bulk
description: Ingest logs in bulk via POST /api/{org}/_bulk using NDJSON. Compatible with Elasticsearch _bulk API and supports up to 200 fields per record.
---

# Logs Ingestion - Bulk

Endpoint: `POST /api/{organization}/_bulk`

This will upload multiple records in batch with ndjson (newline delimited json). This API is compatible with Elasticsearch _bulk API.

## Request

e.g. POST /api/myorg/_bulk

```json
{ "index" : { "_index" : "stream1" } } 
{ "kubernetes.annotations.kubectl.kubernetes.io/default-container": "prometheus", "kubernetes.annotations.kubernetes.io/psp": "eks.privileged", "kubernetes.container_hash": "quay.io/prometheus/prometheus@sha256:4748e26f9369ee7270a7cd3fb9385c1adb441c05792ce2bce2f6dd622fd91d38", "kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1", "kubernetes.container_name": "prometheus", "kubernetes.docker_id": "563f8f40062cd0188c11f39e89d47e6eacddb5624a8a93b39f77ec53b5c38bf5", "kubernetes.host": "ip-10-2-50-35.us-east-2.compute.internal", "kubernetes.labels.app.kubernetes.io/component": "prometheus", "kubernetes.labels.app.kubernetes.io/instance": "k8s", "kubernetes.labels.app.kubernetes.io/managed-by": "prometheus-operator", "kubernetes.labels.app.kubernetes.io/name": "prometheus", "kubernetes.labels.app.kubernetes.io/part-of": "kube-prometheus", "kubernetes.labels.app.kubernetes.io/version": "2.39.1", "kubernetes.labels.controller-revision-hash": "prometheus-k8s-5857d9766c", "kubernetes.labels.operator.prometheus.io/name": "k8s", "kubernetes.labels.operator.prometheus.io/shard": "0", "kubernetes.labels.prometheus": "k8s", "kubernetes.labels.statefulset.kubernetes.io/pod-name": "prometheus-k8s-1", "kubernetes.namespace_name": "monitoring", "kubernetes.pod_id": "ebdc171d-c891-495f-b4d6-e24711b70e64", "kubernetes.pod_name": "prometheus-k8s-1", "log": "ts=2022-12-27T14:09:59.212Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg=\"pkg/mod/k8s.io/client-go@v0.25.1/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User \\\"system:serviceaccount:monitoring:prometheus-k8s\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\"", "stream": "stderr" }
{ "index" : { "_index" : "stream1" } } 
{ "kubernetes.annotations.kubectl.kubernetes.io/default-container": "prometheus", "kubernetes.annotations.kubernetes.io/psp": "eks.privileged", "kubernetes.container_hash": "quay.io/prometheus/prometheus@sha256:4748e26f9369ee7270a7cd3fb9385c1adb441c05792ce2bce2f6dd622fd91d38", "kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1", "kubernetes.container_name": "prometheus", "kubernetes.docker_id": "563f8f40062cd0188c11f39e89d47e6eacddb5624a8a93b39f77ec53b5c38bf5", "kubernetes.host": "ip-10-2-50-35.us-east-2.compute.internal", "kubernetes.labels.app.kubernetes.io/component": "prometheus", "kubernetes.labels.app.kubernetes.io/instance": "k8s", "kubernetes.labels.app.kubernetes.io/managed-by": "prometheus-operator", "kubernetes.labels.app.kubernetes.io/name": "prometheus", "kubernetes.labels.app.kubernetes.io/part-of": "kube-prometheus", "kubernetes.labels.app.kubernetes.io/version": "2.39.1", "kubernetes.labels.controller-revision-hash": "prometheus-k8s-5857d9766c", "kubernetes.labels.operator.prometheus.io/name": "k8s", "kubernetes.labels.operator.prometheus.io/shard": "0", "kubernetes.labels.prometheus": "k8s", "kubernetes.labels.statefulset.kubernetes.io/pod-name": "prometheus-k8s-1", "kubernetes.namespace_name": "monitoring", "kubernetes.pod_id": "ebdc171d-c891-495f-b4d6-e24711b70e64", "kubernetes.pod_name": "prometheus-k8s-1", "log": "ts=2022-12-27T14:09:59.212Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg=\"pkg/mod/k8s.io/client-go@v0.25.1/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \\\"system:serviceaccount:monitoring:prometheus-k8s\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\"", "stream": "stderr" }
```

> First line is stream action
>
> Second line is record data

### Request action

Create record

```json
{ "index" : { "_index" : "stream1" } } 
```

We support `create`, `index`, and `update` actions. `delete` is not supported.

The `_index` is stream name what you want to use.

## Response

The response follows the same shape as the Elasticsearch `_bulk` API, not the simplified `code`/`status` shape shown in older versions of this page:

```json
{
  "took": 0,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "stream1",
        "_id": "5uhtM5kBRQHIfnE6L6mm",
        "_version": 1,
        "result": "created",
        "_shards": { "total": 1, "successful": 1, "failed": 0 },
        "_seq_no": 1,
        "_primary_term": 1,
        "status": 200
      }
    }
  ]
}
```

`items` has one entry per submitted record, each keyed by the action it was submitted with (`index`, `create`, or `update`). `errors` is `true` if any item failed. `took` is always `0` — OpenObserve does not currently populate it with elapsed time.

`_version`, `_shards`, `_seq_no`, and `_primary_term` are fixed placeholder values kept for Elasticsearch API compatibility; they don't carry real per-record metadata.

A failed item has `status` >= 400 and carries `error` and `originalRecord` instead of `result`/`_shards`/`_seq_no`/`_primary_term`:

```json
{
  "index": {
    "_index": "stream1",
    "_id": "5uhtM5kBRQHIfnE6L6mm",
    "status": 422,
    "error": {
      "type": "Too old data, only last 5 hours data can be ingested. Data discarded.",
      "reason": "Too old data, only last 5 hours data can be ingested. Data discarded.",
      "index_uuid": "1",
      "shard": "1",
      "index": "stream1"
    },
    "originalRecord": {
      "kubernetes.container_name": "prometheus",
      "log": "..."
    }
  }
}
```

## Restriction on number of fields/columns per record
> Applicable to cloud version

Please note only records having 200 or less fields/columns will be considered for ingestion , records having more than 200 fields/columns will be discarded with failed status.

> Applicable to open source version

One can configure ZO_COLS_PER_RECORD_LIMIT to set desired value for allowed number of fields/columns per record.


## Timestamp

By default we add a field `_timestamp` for each record with the value of `NOW` in microseconds (unix epoch value). 

we support use of two fields to override the default value.

- _timestamp
- @timestamp

2 data formats are supported for timestamp fields
the value support two data type format:

- microseconds (unix epoch value)
- string value
    - RFC 3339 and ISO 8601 date and time string such as `1996-12-19T16:39:57-08:00`
    - RFC 2822 date and time string such as `Tue, 1 Jul 2003 10:52:37 +0200`

eg:

use microseconds

```json
{ "index" : { "_index" : "stream1" } } 
{ "kubernetes.container_name": "prometheus", "_timestamp": "1674789786006000" }
```

use string datetime

```json
{ "index" : { "_index" : "stream1" } } 
{ "kubernetes.container_name": "prometheus", "_timestamp": "2023-01-02T10:01:01Z" }
```

---
title: Delete
metaTitle: Delete a Stream - OpenObserve API
description: Delete OpenObserve streams via API. Deletion is async and handled by the compactor. Configure auto-deletion with data retention environment settings.
---

## Delete stream
OpenObserve provides multiple deletion strategies to manage your data lifecycle: immediate complete stream deletion, targeted time-range deletion with job tracking, and automatic retention-based cleanup.

## Overview
The Delete Stream API allows you to:

- Delete an entire stream and all its data
- Delete data within a specific time period with job tracking
- Monitor deletion job progress across clusters
- Manage cached query results
All deletion operations are asynchronous and processed by the Compactor service.

## Base URL
`https://example.remote.dev/`
Replace `example.remote.dev` with your actual OpenObserve instance URL.

## Content type
All requests and responses use JSON format.
```
Content-Type: application/json
```
## Endpoints

### Delete entire stream
Delete a complete stream and all associated data.

#### Request
**Method**: `DELETE` <br>
**Path**: `/api/{org_id}/streams/{stream_name}?type=logs&delete_all=true` <br>
**Parameters**:
| Name | Type | Location | Required | Description |
|------|------|----------|----------|-------------|
| org_id | string | path | Yes | Organization identifier |
| stream_name | string | path | Yes | Name of the stream to delete |
| type | string | query | Yes | Stream type: `logs`, `metrics`, or `traces` |
| delete_all | boolean | path | Yes | Delete all related resources like alerts and dashboards |

#### Request example
```bash
curl -X 'DELETE' \
  'https://example.remote.dev/api/default/streams/pii_test?type=logs&delete_all=true' \
  -H 'accept: application/json'
```

#### Response
**Status Code:** `200 OK`
```json
{
  "code": 200,
  "message": "stream deleted"
}
```

#### Response fields
| Field | Type | Description |
|-------|------|-------------|
| code | integer | HTTP status code |
| message | string | Confirmation message |

#### Status codes
| Code | Meaning |
|------|---------|
| 200 | Stream deleted successfully |
| 400 | Invalid parameters |
| 404 | Stream not found |
| 500 | Internal server error |

#### Behavior
Deletion is asynchronous and does not happen immediately:

1. When you call this API, the deletion request is marked in the system.
2. The API responds immediately, you do not wait for actual deletion.
3. A background service called Compactor checks for pending deletions every 10 minutes.
4. When Compactor runs, it starts deleting your stream. This can take anywhere from seconds to several minutes depending on how much data the stream contains.
5. In the worst-case scenario (if you request deletion just before Compactor runs), the entire process could take up to 30 minutes total.
6. You do not need to wait. The deletion happens in the background. You can check the stream status later to confirm it has been deleted.

:::note[Notes]
:::

	- This operation cannot be undone.
	- Data is deleted from both the `file_list` table and object store.
	- No job tracking is available for this endpoint

:::note[Environment variables]
:::
	- You can change the `compactor` run interval: `ZO_COMPACT_INTERVAL=600`. Unit is second. default is `10 minutes`.
	- You can configure data life cycle to auto delete old data: `ZO_COMPACT_DATA_RETENTION_DAYS=30`. The system will auto delete the data after `30` days. Note that the value must be greater than `0`.

### Delete stream data by time range
Delete stream data within a specific time period with job tracking.

#### Request
**Method:** `DELETE`
<br>
**Path:** `/api/{org_id}/streams/{stream_name}/data_by_time_range?start=<start_ts>&end=<end_ts>`

#### Parameters
| Parameter | Type | Location | Description |
|-----------|------|----------|-------------|
| `org_id` | string | Path | Organization identifier |
| `stream_name` | string | Path | Name of the stream |
| `start` | long | path | Start timestamp in microseconds (UTC). Inclusive. |
| `end` | long | path | End timestamp in microseconds (UTC). Inclusive. |
#### Request example
```bash
curl -X DELETE \
  'https://example.remote.dev/api/default/streams/test_stream/data_by_time_range?start=1748736000000000&end=1751241600000000'
```
#### Response
**Status Code:** `200 OK`
```json
{
  "id": "30ernyKEEMznL8KIXEaZhmDYRR9"
}
```
#### Response fields
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Unique job ID for tracking deletion progress |
#### Status codes
| Code | Meaning |
|------|---------|
| 200 | Deletion job created successfully |
| 400 | Invalid parameters (For example, invalid timestamp format) |
| 404 | Stream not found |
#### Behavior
- Initiates a compaction delete job.
- Returns a job ID that can be used to track progress.
- Deletes data from:

    - `file_list` table
    - Object store (for example, S3)
- Granularity:

    - **Logs:** Data is deleted every hour.
    - **Traces:** Data is deleted daily.
---

### Get delete job status
Check the status of a time-range deletion job.

#### Request
**Method:** `GET`
<br>
**Path:** `/api/{org_id}/streams/{stream_name}/data_by_time_range/status/{id}`

#### Parameters
| Parameter | Type | Location | Description |
|-----------|------|----------|-------------|
| `org_id` | string | Path | Organization identifier |
| `stream_name` | string | Path | Name of the stream |
| `id` | string | Path | Job ID returned from deletion request |

#### Request example
```bash
curl -X GET \
  'https://example.remote.dev/api/default/streams/test_stream/data_by_time_range/status/30ernyKEEMznL8KIXEaZhmDYRR9'
```

#### Response: Completed
**Status Code:** `200 OK`
```json
{
    "id": "30f080gLbU4i21VpY2O3YzwrKDH",
    "status": "Completed",
    "metadata": [
        {
            "cluster": "dev3",
            "region": "us-test-3",
            "id": "30f080gLbU4i21VpY2O3YzwrKDH",
            "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z",
            "created_at": 1754003156467113,
            "ended_at": 1754003356516415,
            "status": "Completed"
        },
        {
            "cluster": "dev4",
            "region": "us-test-4",
            "id": "30f080gLbU4i21VpY2O3YzwrKDH",
            "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z",
            "created_at": 1754003156467113,
            "ended_at": 1754003326523177,
            "status": "Completed"
        }
    ]
}
```
#### Response: Pending
**Status Code:** `200 OK`
```json
{
    "id": "30f080gLbU4i21VpY2O3YzwrKDH",
    "status": "Pending",
    "metadata": [
        {
            "cluster": "dev3",
            "region": "us-test-3",
            "id": "30f080gLbU4i21VpY2O3YzwrKDH",
            "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z",
            "created_at": 1754003156467113,
            "ended_at": 0,
            "status": "Pending"
        },
        {
            "cluster": "dev4",
            "region": "us-test-4",
            "id": "30f080gLbU4i21VpY2O3YzwrKDH",
            "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z",
            "created_at": 1754003156467113,
            "ended_at": 0,
            "status": "Pending"
        }
    ]
}
```
#### Response: With Errors
**Status Code:** `200 OK`
```json
{
    "id": "30fCWBSNWwTWnRJE0weFfDIc3zz",
    "status": "Pending",
    "metadata": [
        {
            "cluster": "dev4",
            "region": "us-test-4",
            "id": "30fCWBSNWwTWnRJE0weFfDIc3zz",
            "key": "default/logs/delete_d4/2025-07-21T14:00:00Z,2025-07-22T00:00:00Z",
            "created_at": 1754009269552227,
            "ended_at": 1754009558553845,
            "status": "Completed"
        }
    ],
    "errors": [
        {
            "cluster": "dev3",
            "error": "Error getting delete job status from cluster node: Status { code: Internal, message: \"Database error: DbError# SeaORMError# job not found\", metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Fri, 01 Aug 2025 00:58:01 GMT\", \"content-length\": \"0\"} }, source: None }",
            "region": "us-test-3"
        }
    ]
}
```
#### Response fields
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Job identifier |
| `status` | string | Overall job status: `Completed` or `Pending` |
| `metadata` | array | Array of per-cluster deletion details |
| `metadata[].cluster` | string | Cluster identifier |
| `metadata[].region` | string | Region/zone identifier |
| `metadata[].id` | string | Job ID |
| `metadata[].key` | string | Database key for the deletion operation |
| `metadata[].created_at` | long | Job creation timestamp in microseconds |
| `metadata[].ended_at` | long | Job completion timestamp in microseconds (0 if still pending) |
| `metadata[].status` | string | Individual cluster deletion status |
| `errors` | array | Array of errors from specific clusters (if any) |
| `errors[].cluster` | string | Cluster where error occurred |
| `errors[].region` | string | Region identifier |
| `errors[].error` | string | Error message |

#### Status Codes
| Code | Meaning |
|------|---------|
| 200 | Status retrieved successfully |
| 404 | Job ID not found |

#### Behavior
- Returns current status of deletion job
- Shows progress across all clusters in distributed setup
- Shows error details if any cluster encountered failures
- Status of `Pending` means deletion is still in progress
- Status of `Completed` means all clusters finished deletion
---

### Delete cache results
Delete cached query results for a stream.
#### Request
**Method:** `DELETE`
<br>
**Path:** `/api/{org_id}/streams/{stream_name}/cache/results?type=<stream_type>&ts=<timestamp>`

### Parameters
| Parameter | Type | Location | Description |
|-----------|------|----------|-------------|
| `org_id` | string | path | Organization identifier |
| `stream_name` | string | Path | Stream name (use `_all` to delete cache for all streams) |
| `type` | string | Query | Stream type: `logs`, `metrics`, or `traces` |
| `ts` | long | Query | Timestamp threshold in microseconds. Deletes cache from start up to this timestamp. Retains cache from timestamp onwards. |

#### Request example
```bash
curl -X DELETE \
  'https://example.remote.dev/api/default/streams/test_stream/_all/cache/results?type=logs&ts=1753849800000'
```

### Response
**Status Code:** `200 OK`
```json
{
  "code": 200,
  "message": "cache deleted"
}
```

### Response Fields
| Field | Type | Description |
|-------|------|-------------|
| `code` | integer | HTTP status code |
| `message` | string | Confirmation message |

### Status Codes
| Code | Meaning |
|------|---------|
| 200 | Cache deleted successfully |
| 400 | Invalid parameters |
| 404 | Stream not found |

### Behavior
- Accepts `ts` (timestamp) query parameter in microseconds
- Deletes cache from `cache_start` up to the given `ts`
- Retains cache from `ts` onwards

---
title: Overview
metaTitle: API Reference | OpenObserve
description: Programmatic access to OpenObserve. HTTP basic-auth endpoints for streams, ingestion, search, functions, users, and metrics.
---

# API Index

These APIs can be used to programmatically interact with OpenObserve.

All APIs must have an authorization header. Authorization header can be created using base64 encoded values of user id and password. For the sake of simplicity it is HTTP basic authentication mechanism.

Header creation mechanism:

```
Authorization: Basic base64("username:password")
```

For example:

```
Authorization: Basic YWRtaW46Q29tcGxleHBhc3MjMTIz
```

Make sure that you are sending the requests over HTTPS.

## API List

1. [Stream](stream/index.md)
    1. [List](stream/list.md)
    1. [Schema](stream/schema.md)
    1. [Setting](stream/setting.md)
1. [Ingestion](ingestion/index.md)
    1. [Bulk](ingestion/logs/bulk.md)
    1. [Json](ingestion/logs/json.md)
    1. [Multi](ingestion/logs/multi.md)
1. [Search](search/index.md)
1. [Function](function/index.md)
1. [User](user/index.md)
    1. [Create](user/create.md)
    1. [Delete](user/delete.md)
    1. [List](user/list.md)
1. [Metrics](metrics.md)
1. [Cluster](https://openobserve.ai/docs/reference/api/cluster/cluster-info/)

## Next steps

- [Quickstart](../../getting-started.md): get OpenObserve running and grab your credentials.
- [Ingestion](../../ingestion/index.md): start sending logs, metrics, and traces.
- [OpenTelemetry / OTLP](../../ingestion/logs/otlp.md): the recommended modern ingestion path.

**Need some help?**

- Join our [Community Slack](https://short.openobserve.ai/community) 
- Or [Contact support](https://openobserve.ai/contactus/)

---
title: JSON
metaTitle: Ingest Logs as JSON - OpenObserve API
description: Ingest logs in batch via POST /api/{org}/{stream}/_json using standard JSON. Supports flattening, timestamps, and up to 200 fields per record.
---

# Logs Ingestion - JSON

Endpoint: `POST /api/{organization}/{stream}/_json`



This will upload multiple records in batch with standard json format.

## Request 

e.g. `POST /api/myorg/stream1/_json`

```json
[
    {
        "kubernetes.annotations.kubectl.kubernetes.io/default-container": "prometheus",
        "kubernetes.annotations.kubernetes.io/psp": "eks.privileged",
        "kubernetes.container_hash": "quay.io/prometheus/prometheus@sha256:4748e26f9369ee7270a7cd3fb9385c1adb441c05792ce2bce2f6dd622fd91d38",
        "kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1",
        "kubernetes.container_name": "prometheus",
        "kubernetes.docker_id": "563f8f40062cd0188c11f39e89d47e6eacddb5624a8a93b39f77ec53b5c38bf5",
        "kubernetes.host": "ip-10-2-50-35.us-east-2.compute.internal",
        "kubernetes.labels.app.kubernetes.io/component": "prometheus",
        "kubernetes.labels.app.kubernetes.io/instance": "k8s",
        "kubernetes.labels.app.kubernetes.io/managed-by": "prometheus-operator",
        "kubernetes.labels.app.kubernetes.io/name": "prometheus",
        "kubernetes.labels.app.kubernetes.io/part-of": "kube-prometheus",
        "kubernetes.labels.app.kubernetes.io/version": "2.39.1",
        "kubernetes.labels.controller-revision-hash": "prometheus-k8s-5857d9766c",
        "kubernetes.labels.operator.prometheus.io/name": "k8s",
        "kubernetes.labels.operator.prometheus.io/shard": "0",
        "kubernetes.labels.prometheus": "k8s",
        "kubernetes.labels.statefulset.kubernetes.io/pod-name": "prometheus-k8s-1",
        "kubernetes.namespace_name": "monitoring",
        "kubernetes.pod_id": "ebdc171d-c891-495f-b4d6-e24711b70e64",
        "kubernetes.pod_name": "prometheus-k8s-1",
        "log": "ts=2022-12-27T14:09:59.212Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg=\"pkg/mod/k8s.io/client-go@v0.25.1/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User \\\"system:serviceaccount:monitoring:prometheus-k8s\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\"",
        "stream": "stderr"
    },
		{
        "kubernetes.annotations.kubectl.kubernetes.io/default-container": "prometheus",
        "kubernetes.annotations.kubernetes.io/psp": "eks.privileged",
        "kubernetes.container_hash": "quay.io/prometheus/prometheus@sha256:4748e26f9369ee7270a7cd3fb9385c1adb441c05792ce2bce2f6dd622fd91d38",
        "kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1",
        "kubernetes.container_name": "prometheus",
        "kubernetes.docker_id": "563f8f40062cd0188c11f39e89d47e6eacddb5624a8a93b39f77ec53b5c38bf5",
        "kubernetes.host": "ip-10-2-50-35.us-east-2.compute.internal",
        "kubernetes.labels.app.kubernetes.io/component": "prometheus",
        "kubernetes.labels.app.kubernetes.io/instance": "k8s",
        "kubernetes.labels.app.kubernetes.io/managed-by": "prometheus-operator",
        "kubernetes.labels.app.kubernetes.io/name": "prometheus",
        "kubernetes.labels.app.kubernetes.io/part-of": "kube-prometheus",
        "kubernetes.labels.app.kubernetes.io/version": "2.39.1",
        "kubernetes.labels.controller-revision-hash": "prometheus-k8s-5857d9766c",
        "kubernetes.labels.operator.prometheus.io/name": "k8s",
        "kubernetes.labels.operator.prometheus.io/shard": "0",
        "kubernetes.labels.prometheus": "k8s",
        "kubernetes.labels.statefulset.kubernetes.io/pod-name": "prometheus-k8s-1",
        "kubernetes.namespace_name": "monitoring",
        "kubernetes.pod_id": "ebdc171d-c891-495f-b4d6-e24711b70e64",
        "kubernetes.pod_name": "prometheus-k8s-1",
        "log": "ts=2022-12-27T14:09:59.212Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg=\"pkg/mod/k8s.io/client-go@v0.25.1/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \\\"system:serviceaccount:monitoring:prometheus-k8s\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\"",
        "stream": "stderr"
    }
]
```

Each line is one record.

## Response

```json
{
	"code": 200,
	"status": [
		{
			"name": "stream1",
			"successful": 2,
			"failed": 0
		}
	]
}
```

Returns successful and failed count for each stream.

## Restriction on number of fields/columns per record
> Applicable to cloud version

Please note only records having 200 or less fields/columns will be considered for ingestion , records having more than 200 fields/columns will be discarded with failed status.

> Applicable to open source version

One can configure ZO_COLS_PER_RECORD_LIMIT to set desired value for allowed number of fields/columns per record.

## Flattening of the JSON structure

OpenObserve flattens deep JSON logs. Below is an example log before and after being flattened.

### Before

```json
{
	"actor": {
		"ip": "[redacted]",
		"id": 558875,
		"parent" : {
			"id": 45516,
			"active": true
		}
	}
	"response": {
		"error_occured": false,
		"status_code": 200
	}
	
}
```

### After

```json
{
	"actor_ip": "[redacted]",
	"actor_id": 558875,
	"actor_parent_id": 45516,
	"actor_parent_active": true,
	"response_error_occured": false,
	"response_status_code": 200
}
```

### Restriction on flattening depth

⚠️ For performance reasons, OpenObserve limits the depth at which the JSON structure gets flattened. Past that limit, the generated field will contain unparsed JSON as a string.
The default depth is `3`, but this limit can be configured via the `ZO_INGEST_FLATTEN_LEVEL` environment variable. `ZO_INGEST_FLATTEN_LEVEL` can either be `0`, which disables the flattening limit, or any positive number, to change the depth at which the flattening stops.

## Timestamp

By default we add a field `_timestamp` for each record with the value of `NOW` in microseconds (unix epoch value). 

we support use of two fields to override the default value.

- _timestamp
- @timestamp

2 data formats are supported for timestamp fields
the value support two data type format:

- microseconds (unix epoch value)
- string value
    - RFC 3339 and ISO 8601 date and time string such as `1996-12-19T16:39:57-08:00`
    - RFC 2822 date and time string such as `Tue, 1 Jul 2003 10:52:37 +0200`

eg:

use microseconds

```json
[{
	"kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1",
	"kubernetes.container_name": "prometheus", 
	"_timestamp": "1674789786006000"
}]
```

use string datetime

```json
[{
	"kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1",
	"kubernetes.container_name": "prometheus", 
	"_timestamp": "2023-01-02T10:01:01Z"
}]
```
```

---
title: List
metaTitle: List Streams - OpenObserve API
description: List all streams in OpenObserve by type (logs, metrics, traces). Optionally include schema, storage stats, and stream settings in the response.
---

# List streams

Endpoint: `GET /api/{organization}/streams?fetchSchema=false&type={StreamType}`

## Request

- fetchSchema: true / false

	fetchSchema set to `true` will response the schema for each stream or without schema.

- type: logs / metrics / traces

	default is `logs`.

## Response

```json
{
	"list": [
		{
			"name": "k8s",
			"storage_type": "s3",
			"stream_type": "logs",
			"stats": {
				"doc_time_min": 1673715046856933,
				"doc_time_max": 1673849134852901,
				"doc_num": 3300000,
				"file_num": 16,
				"storage_size": 3323.5,
				"compressed_size": 11.42
			},
            "schema": [
				{
					"name": "_timestamp",
					"type": "Int64"
				},
				{
					"name": "kubernetes.annotations.kubernetes.io/psp",
					"type": "Utf8"
				},
            ],
			"settings": {
				"partition_keys": {},
				"full_text_search_keys": ["log"]
			}
		}
	]
}
```

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| name       | string    | -             | stream name |
| storage_type | string  | -             | s3 / disk   |
| stream_type | string   | logs          | logs / metrics / traces |
| stats      | object    | -             | stats for the stream |
| stats.doc_time_min | int64 | 0         | the minimum timestamp of the record in the stream |
| stats.doc_time_max | int64 | 0         | the maximum timestamp of the record in the stream |
| stats.doc_num      | int64 | 0         | the records num of the stream |
| stats.file_num     | int64 | 0         | the files num in storage of the stream |
| stats.storage_size | int64 | 0         | ingestion data size of the original data |
| stats.compressed_size | int64 | 0      | stored size in storage after compression |
| schema     | array     | -             | the schema of the stream, if `fetchSchema` set to false, has no this field |
| schema.name | string   | -             | field name |
| schema.type | string   | -             | field data type: Utf8 / Int64 / Float64 / Timestamp / Boolean |
| settings   | object    | -             | settings of the stream |
| settings.partition_keys | object | -   | custom partition keys for the stream. By default OpenObserve uses timestamp as the first level partition key |
| settings.full_text_search_keys | array[string] | - | full text search fields, default OpenObserve uses `log`, `message`, `msg`, `content`, `data`, `json`, if there is no those fields in your stream, will report error: `you should set the full text search fields`. |

---
title: Loki
description: "Ingest logs through the Grafana Loki-compatible push API. Supports Loki stream labels, nanosecond timestamps, and structured metadata."
---

# Logs Ingestion - Loki

Endpoint: `POST /api/{organization}/loki/api/v1/push`

OpenObserve is compatible with the Grafana Loki push API. You can send logs using any Loki-compatible client (e.g. Promtail, Grafana Agent, Alloy) by pointing it at OpenObserve.

> we use the `o2_stream_name` label (or the legacy `stream_name` label) for custom stream name, default will push into `default` stream.

## Request

e.g. `POST /api/myorg/loki/api/v1/push`

Content-Type: `application/json`

```json
{
  "streams": [
    {
      "stream": {
        "o2_stream_name": "custom_stream",
        "kubernetes_namespace": "monitoring",
        "kubernetes_pod_name": "prometheus-k8s-1",
        "kubernetes_container_name": "prometheus"
      },
      "values": [
        [
          "1672149599212000000",
          "ts=2022-12-27T14:09:59.212Z caller=klog.go:108 level=warn component=k8s_client_runtime msg=\"failed to list *v1.Pod\""
        ],
        [
          "1672149600000000000",
          "ts=2022-12-27T14:10:00.000Z caller=klog.go:116 level=error component=k8s_client_runtime msg=\"Failed to watch *v1.Pod\""
        ]
      ]
    }
  ]
}
```

### Fields

| Field | Type | Description |
|-------|------|-------------|
| `streams` | array | List of log streams to push. |
| `streams[].stream` | object | Key-value label pairs that identify the stream. Labels are indexed and can be used for filtering. Use the `o2_stream_name` label (or the legacy `stream_name` label) to set a custom target stream; if neither is present, logs are pushed into the `default` stream. |
| `streams[].values` | array | List of log entries. Each entry is a two-element array: `[timestamp, line]`. |
| `streams[].values[][0]` | string | Unix timestamp in **nanoseconds** as a string. |
| `streams[].values[][1]` | string | Log line content. |

## Response

```json
{}
```

An empty JSON object `{}` with HTTP status `204 No Content` indicates success, matching the standard Loki push API behavior.

## Authentication

Pass your credentials using HTTP Basic Auth or via the `Authorization` header, the same as all other OpenObserve ingestion endpoints.

```
Authorization: Basic <base64(user:password)>
```

## Configuring Promtail

Point Promtail at OpenObserve by setting the Loki push URL in your `promtail.yaml`:

```yaml
clients:
  - url: https://<openobserve-host>/api/<organization>/loki/api/v1/push
    basic_auth:
      username: <user>
      password: <password>
```

## Configuring Grafana Alloy

```alloy
loki.write "openobserve" {
  endpoint {
    url = "https://<openobserve-host>/api/<organization>/loki/api/v1/push"
    basic_auth {
      username = "<user>"
      password = "<password>"
    }
  }
}
```

---
title: Multi
description: Ingest multiple log records in JSON lines via POST /api/{org}/{stream}/_multi. Supports batch upload, timestamps, and up to 200 fields per record.
---

# Logs Ingestion - Multi

Endpoint: `POST /api/{organization}/{stream}/_multi`

This will upload multiple records in batch with multiple json lines.

## Request

e.g. `POST /api/myorg/stream1/_multi`

```json
{ "kubernetes.annotations.kubectl.kubernetes.io/default-container": "prometheus", "kubernetes.annotations.kubernetes.io/psp": "eks.privileged", "kubernetes.container_hash": "quay.io/prometheus/prometheus@sha256:4748e26f9369ee7270a7cd3fb9385c1adb441c05792ce2bce2f6dd622fd91d38", "kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1", "kubernetes.container_name": "prometheus", "kubernetes.docker_id": "563f8f40062cd0188c11f39e89d47e6eacddb5624a8a93b39f77ec53b5c38bf5", "kubernetes.host": "ip-10-2-50-35.us-east-2.compute.internal", "kubernetes.labels.app.kubernetes.io/component": "prometheus", "kubernetes.labels.app.kubernetes.io/instance": "k8s", "kubernetes.labels.app.kubernetes.io/managed-by": "prometheus-operator", "kubernetes.labels.app.kubernetes.io/name": "prometheus", "kubernetes.labels.app.kubernetes.io/part-of": "kube-prometheus", "kubernetes.labels.app.kubernetes.io/version": "2.39.1", "kubernetes.labels.controller-revision-hash": "prometheus-k8s-5857d9766c", "kubernetes.labels.operator.prometheus.io/name": "k8s", "kubernetes.labels.operator.prometheus.io/shard": "0", "kubernetes.labels.prometheus": "k8s", "kubernetes.labels.statefulset.kubernetes.io/pod-name": "prometheus-k8s-1", "kubernetes.namespace_name": "monitoring", "kubernetes.pod_id": "ebdc171d-c891-495f-b4d6-e24711b70e64", "kubernetes.pod_name": "prometheus-k8s-1", "log": "ts=2022-12-27T14:09:59.212Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg=\"pkg/mod/k8s.io/client-go@v0.25.1/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User \\\"system:serviceaccount:monitoring:prometheus-k8s\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\"", "stream": "stderr" }
{ "kubernetes.annotations.kubectl.kubernetes.io/default-container": "prometheus", "kubernetes.annotations.kubernetes.io/psp": "eks.privileged", "kubernetes.container_hash": "quay.io/prometheus/prometheus@sha256:4748e26f9369ee7270a7cd3fb9385c1adb441c05792ce2bce2f6dd622fd91d38", "kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1", "kubernetes.container_name": "prometheus", "kubernetes.docker_id": "563f8f40062cd0188c11f39e89d47e6eacddb5624a8a93b39f77ec53b5c38bf5", "kubernetes.host": "ip-10-2-50-35.us-east-2.compute.internal", "kubernetes.labels.app.kubernetes.io/component": "prometheus", "kubernetes.labels.app.kubernetes.io/instance": "k8s", "kubernetes.labels.app.kubernetes.io/managed-by": "prometheus-operator", "kubernetes.labels.app.kubernetes.io/name": "prometheus", "kubernetes.labels.app.kubernetes.io/part-of": "kube-prometheus", "kubernetes.labels.app.kubernetes.io/version": "2.39.1", "kubernetes.labels.controller-revision-hash": "prometheus-k8s-5857d9766c", "kubernetes.labels.operator.prometheus.io/name": "k8s", "kubernetes.labels.operator.prometheus.io/shard": "0", "kubernetes.labels.prometheus": "k8s", "kubernetes.labels.statefulset.kubernetes.io/pod-name": "prometheus-k8s-1", "kubernetes.namespace_name": "monitoring", "kubernetes.pod_id": "ebdc171d-c891-495f-b4d6-e24711b70e64", "kubernetes.pod_name": "prometheus-k8s-1", "log": "ts=2022-12-27T14:09:59.212Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg=\"pkg/mod/k8s.io/client-go@v0.25.1/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \\\"system:serviceaccount:monitoring:prometheus-k8s\\\" cannot list resource \\\"pods\\\" in API group \\\"\\\" at the cluster scope\"", "stream": "stderr" }
```

Each line is one json record.

## Response

```json
{
	"code": 200,
	"status": [
		{
			"name": "stream1",
			"successful": 2,
			"failed": 0
		}
	]
}
```

Returns successful and failed count for each stream.

## Restriction on number of fields/columns per record
> Applicable to cloud version

Please note only records having 200 or less fields/columns will be considered for ingestion , records having more than 200 fields/columns will be discarded with failed status.

> Applicable to open source version

One can configure ZO_COLS_PER_RECORD_LIMIT to set desired value for allowed number of fields/columns per record.

## Timestamp

By default we add a field `_timestamp` for each record with the value of `NOW` in microseconds (unix epoch value). 

we support use of two fields to override the default value.

- _timestamp
- @timestamp

2 data formats are supported for timestamp fields
the value support two data type format:

- microseconds (unix epoch value)
- string value
    - RFC 3339 and ISO 8601 date and time string such as `1996-12-19T16:39:57-08:00`
    - RFC 2822 date and time string such as `Tue, 1 Jul 2003 10:52:37 +0200`

eg:

use microseconds

```json
{ "kubernetes.container_name": "prometheus", "_timestamp": "1674789786006000" }
```

use string datetime

```json
{ "kubernetes.container_name": "prometheus", "_timestamp": "2023-01-02T10:01:01Z" }
```

---
title: Otlp
description: Ingest logs via OpenTelemetry Protocol (OTLP) at POST /api/{org}/v1/logs. Supports OTLP JSON and Protobuf formats with resource attributes and log records.
---

# Logs Ingestion - OTLP

Endpoint: `POST /api/{organization}/v1/logs`

OpenObserve supports the OpenTelemetry Protocol (OTLP) for log ingestion. You can send logs from any OpenTelemetry-compatible collector or SDK by pointing it at OpenObserve.

> we use custom http header `stream-name` for speciafic stream name, default will push into `default` stream.

## Request

e.g. `POST /api/myorg/v1/logs`

Content-Type: `application/json` (JSON) or `application/x-protobuf` (Protobuf)

```json
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "my-service" }
          },
          {
            "key": "service.version",
            "value": { "stringValue": "1.2.3" }
          },
          {
            "key": "host.name",
            "value": { "stringValue": "ip-10-2-50-35.us-east-2.compute.internal" }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my-logger",
            "version": "1.0.0"
          },
          "logRecords": [
            {
              "timeUnixNano": "1672149599212000000",
              "observedTimeUnixNano": "1672149599212000000",
              "severityNumber": 9,
              "severityText": "INFO",
              "body": {
                "stringValue": "Request processed successfully"
              },
              "attributes": [
                {
                  "key": "http.method",
                  "value": { "stringValue": "GET" }
                },
                {
                  "key": "http.status_code",
                  "value": { "intValue": "200" }
                },
                {
                  "key": "trace_id",
                  "value": { "stringValue": "4bf92f3577b34da6a3ce929d0e0e4736" }
                }
              ],
              "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
              "spanId": "00f067aa0ba902b7"
            }
          ]
        }
      ]
    }
  ]
}
```

### Fields

| Field | Type | Description |
|-------|------|-------------|
| `resourceLogs` | array | List of resource log groups. |
| `resourceLogs[].resource` | object | Resource describing the entity producing the logs (e.g. service, host). |
| `resourceLogs[].resource.attributes` | array | Key-value pairs for resource-level metadata. |
| `resourceLogs[].scopeLogs` | array | List of instrumentation scope log groups. |
| `scopeLogs[].scope` | object | Instrumentation scope (library name and version). |
| `scopeLogs[].logRecords` | array | List of individual log records. |
| `logRecords[].timeUnixNano` | string | Log timestamp in **nanoseconds** since Unix epoch. |
| `logRecords[].observedTimeUnixNano` | string | Time the log was observed by the collector, in nanoseconds. |
| `logRecords[].severityNumber` | integer | Numeric severity level (1–24). See [OTLP severity levels](https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitynumber). |
| `logRecords[].severityText` | string | Human-readable severity string (e.g. `INFO`, `WARN`, `ERROR`). |
| `logRecords[].body` | object | Log message body. Typically a `stringValue`. |
| `logRecords[].attributes` | array | Key-value pairs for log-level metadata. |
| `logRecords[].traceId` | string | Trace ID associated with this log record (hex string). |
| `logRecords[].spanId` | string | Span ID associated with this log record (hex string). |

## Response

```json
{
  "partialSuccess": {}
}
```

HTTP status `200 OK` with an empty `partialSuccess` object indicates all records were accepted, matching the standard OTLP HTTP response format.

## Authentication

Pass your credentials using HTTP Basic Auth or via the `Authorization` header:

```
Authorization: Basic <base64(user:password)>
```

## Configuring OpenTelemetry Collector

Configure the OTLP exporter in your OpenTelemetry Collector `config.yaml` to forward logs to OpenObserve:

```yaml
exporters:
  otlphttp/openobserve:
    endpoint: https://<openobserve-host>/api/<organization>
    headers:
      Authorization: "Basic <base64(user:password)>"
      stream-name: "custom_stream"

service:
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [otlphttp/openobserve]
```

---
title: Schema
description: Retrieve schema, stats, and settings for a log, metric, or trace stream in OpenObserve via GET API. Includes timestamps, field types, and storage info.
---

# Get schema for stream

Endpoint: `GET /api/{organization}/streams/{stream}/schema?type={StreamType}`

## Request

- type: logs / metrics / traces

    default is `logs`.

## Response

```json
{
    "name": "k8s",
    "storage_type": "s3",
    "stream_type": "logs",
    "stats": {
        "doc_time_min": 1673715046856933,
        "doc_time_max": 1673849134852901,
        "doc_num": 3300000,
        "file_num": 16,
        "storage_size": 3323.5,
        "compressed_size": 11.42
    },
    "schema": [
        {
            "name": "_timestamp",
            "type": "Int64"
        },
        {
            "name": "kubernetes.annotations.kubernetes.io/psp",
            "type": "Utf8"
        },
    ],
    "settings": {
        "partition_keys": {},
        "full_text_search_keys": ["log"]
    }
}
```

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| name       | string    | -             | stream name |
| storage_type | string  | -             | s3 / disk   |
| stream_type | string   | logs          | logs / metrics / traces |
| stats      | object    | -             | stats for the stream |
| stats.doc_time_min | int64 | 0         | the minimum timestamp of the record in the stream |
| stats.doc_time_max | int64 | 0         | the maximum timestamp of the record in the stream |
| stats.doc_num      | int64 | 0         | the records num of the stream |
| stats.file_num     | int64 | 0         | the files num in storage of the stream |
| stats.storage_size | int64 | 0         | ingestion data size of the original data |
| stats.compressed_size | int64 | 0      | stored size in storage after compression |
| schema     | array     | -             | the schema of the stream, if `fetchSchema` set to false, has no this field |
| schema.name | string   | -             | field name |
| schema.type | string   | -             | field data type: Utf8 / Int64 / Float64 / Timestamp / Boolean |
| settings   | object    | -             | settings of the stream |
| settings.partition_keys | object | -   | custom partition keys for the stream. By default OpenObserve uses timestamp as the first level partition key |
| settings.full_text_search_keys | array[string] | - | full text search fields, default OpenObserve uses `log`, `message`, `msg`, `content`, `data`, `json`, if there is no those fields in your stream, will report error: `you should set the full text search fields`. |

---
title: Search
metaTitle: Search API | OpenObserve
description: Search logs with SQL using POST /api/{org}/_search. Filter by time, size, and conditions. Supports full text, aggregations, and custom functions.
---

# Search

Endpoint: `POST /api/{organization}/_search`

Replace `{stream}` in the SQL examples below with your stream name (e.g. `default`).

## Request

```json
{
    "query": {
        "sql": "SELECT * FROM {stream} WHERE [condition]",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000,
        "from": 0,
        "size": 0
    },
    "search_type": "ui",
    "timeout": 0
}
```

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| query      | object    | -             | query params |
| query.sql  | string    | -             | use SQL query data, and filter data by `start_time` and `end_time`, and default order by _timestamp, you can use order by override order, and fetch offset limit by `form` and `size` |
| query.start_time | int64 | 0           | unit: microseconds, filter data by time range, you need always provide this value |
| query.end_time   | int64 | 0           | unit: microseconds, filter data by time range, you need always provide this value |
| query.from | int64     | 0             | offset in SQL |
| query.size | int64     | 0             | limit in SQL  |
| search_type | string   | -             | default is empty, support: `ui`, `dashboards`, `reports`, `alerts` |
| timeout    | int       | 0             | default value based on `ZO_QUERY_TIMEOUT=600` |
| agent_options | object    | -             | options for agent/MCP clients |
| agent_options.mode | string | `default` | query execution mode: `default` (result cache path) or `partition` (streaming partition loop, see below) |
| agent_options.output_format | string | `json` | response format for hits: `json`, `csv`, or `md_table` |

## Response

```json
{
	"took": 155,
	"hits": [
		{
			"_p": "F",
			"_timestamp": 1674213225158000,
			"kubernetes": {
				"annotations": {
					"kubernetes": {
						"io/psp": "eks.privileged"
					}
				},
				"container_hash": "dkr.ecr.us-west-2.amazonaws.com/ziox@sha256:3dbbb0dc1eab2d5a3b3e4a75fd87d194e8095c92d7b2b62e7cdbd07020f54589",
				"container_image": "dkr.ecr.us-west-2.amazonaws.com/ziox:v0.0.3",
				"container_name": "ziox",
				"docker_id": "eb0983bdb9ff9360d227e6a0b268fe3b24a0868c2c2d725a1516c11e88bf5789",
				"host": "ip.us-east-2.compute.internal",
				"labels": {
					"app": "ziox",
					"controller-revision-hash": "ziox-ingester-579b7767cf",
					"name": "ziox-ingester",
					"role": "ingester",
					"statefulset": {
						"kubernetes": {
							"io/pod-name": "ziox-ingester-0"
						}
					}
				},
				"namespace_name": "ziox",
				"pod_id": "35a0421f-9203-4d73-9663-9ff0ce26d409",
				"pod_name": "ziox-ingester-0"
			},
			"log": "[2023-01-20T11:13:45Z INFO  actix_web::middleware::logger] 10.2.80.192 \"POST /api/demo/_bulk HTTP/1.1\" 200 68 \"-\" \"go-resty/2.7.0 (https://github.com/go-resty/resty)\" 0.001074",
			"stream": "stderr"
		}
	],
	"total": 27179431,
	"from": 0,
	"size": 1,
	"scan_size": 28943
}
```

Response description:

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| took       | int64     | -             | unit: milliseconds, query execute time |
| from       | int64     | 0             | value from `query.from` |
| size       | int64     | 0             | value from `query.size` |
| scan_size  | int64     | 0             | unit: MB, it response the data size scale when execute the query. |
| hits       | array     | -             | records for query, each record is a log row what you ingested. |


## Agent options

### `output_format`

Controls how hits are formatted in the response. Useful for agent/MCP clients that pay per token.

| Value       | Description |
|-------------|-------------|
| `json`      | Default. Hits returned as a JSON array of objects. |
| `csv`       | Tabular hits as a compact CSV block (~40% fewer tokens than JSON). |
| `md_table`  | Tabular hits as a Markdown table. Best for small result sets. |

### `mode: partition`

Set `agent_options.mode` to `partition` to run search through the streaming backend pipeline. The server scans time partitions one by one and stops early once enough rows are collected, which reduces data scanned for top-N queries. For aggregation queries, the server accumulates reusable cache partition by partition, so repeated queries against shifting time windows hit progressively warmer cache instead of invalidating a monolithic entry.

This mode is designed for agent/MCP clients that speak plain request-response and cannot consume SSE — it exposes the same partitioned execution that the streaming `_search_stream` endpoint uses, collected into a single response.

| Value       | Description |
|-------------|-------------|
| `default`   | (default) Single search through the result cache path. Behavior is byte-for-byte unchanged. |
| `partition` | Partitioned execution with per-partition early termination and streaming-agg caching, collected into one response. |

**Key behaviors in partition mode:**

- **Early termination**: scanning stops once the query has enough rows; useful for `SELECT ... LIMIT 100`-style queries scanning a wide time range.
- **Progressive aggregation cache**: aggregation queries (histogram, term counts, etc.) build up the streaming-agg cache partition by partition, so follow-up queries with similar time windows reuse cached work.
- **Cancellation**: if the HTTP client disconnects (drops the request), the per-partition loop stops — you are not billed for scanning partitions you never see.
- **No SSE required**: the response is a plain JSON `Response` identical in shape to `default` mode, with all hit pages and scan counters folded together.

**When to use partition mode:**

- Your client cannot consume SSE (MCP tools, REST clients, curl-based scripts).
- You are scanning a large time range but only need a few rows (top-N or sample).
- You run exploratory queries that shift their time window with each call — the progressive cache rewards this pattern.
- **Do NOT** split the time range yourself and call `_search` in a loop. Let the server do it with one call in `partition` mode.

## SQL Syntax

Please refer to [PostgreSQL](https://www.postgresql.org/docs/current/sql-syntax.html) for SQL Syntax.

Notes:

- We have a build-in time field, `_timestamp` you can use it to do time range filter.
- Field name can not start with `@`.
- Field name can use double quote or without quote.
- Field integer value without quote.
- Field string value must use single quote.

## Limitation

- You should give a time range for each query or it will scan all data, it is a very expensive operate.


## Examples

Here list some common examples, if you want more example please create a issue tell us, we will add it.

### Query latest 10 record logs

```json
{
    "query": {
        "sql": "SELECT * FROM {stream}",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000,
        "from": 0,
        "size": 10
    }
}
```

### Query latest 10 record logs with filter

```json
{
    "query": {
        "sql": "SELECT * FROM {stream} WHERE kubernetes.namespace_name='default' AND code=200 ",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000,
        "from": 0,
        "size": 10
    }
}
```

### Full text query

```json
{
    "query": {
        "sql": "SELECT * FROM {stream} WHERE match_all('err') ",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000,
        "from": 0,
        "size": 10
    }
}
```

### Match on a field (log)

```json
{
    "query": {
        "sql": "SELECT * FROM {stream} WHERE str_match(log, 'err') ",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000,
        "from": 0,
        "size": 10
    }
}
```

### Histogram aggregation (full mode)

```json
{
    "query": {
       "sql": "SELECT histogram(_timestamp, '5 minute') AS key, COUNT(*) AS num FROM {stream} GROUP BY key ORDER BY key LIMIT 10 OFFSET 1",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000
    }
}
```

### Term aggregation (full mode)

```json
{
    "query": {
       "sql": "SELECT kubernetes.namespace_name AS namespace, COUNT(*) AS num FROM {stream} GROUP BY namespace ORDER BY namespace",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000
    }
}
```

### Use custom functions 

```json
{
    "query": {
       "sql": "SELECT *, my_func(log) as mykey FROM {stream}",
        "start_time": 1674789786006000,
        "end_time": 1674789786006000,
        "from": 0,
        "size": 10
    }
}
```

### Partition mode with CSV output

Run a top-N query across a wide time window with partition mode, returning hits as compact CSV:

```json
{
    "query": {
        "sql": "SELECT * FROM {stream} WHERE code=500 ORDER BY _timestamp DESC",
        "start_time": 1674000000000000,
        "end_time": 1675000000000000,
        "from": 0,
        "size": 50
    },
    "agent_options": {
        "mode": "partition",
        "output_format": "csv"
    }
}
```

With `mode: partition`, the server scans partitions one by one and stops early once it has 50 rows, instead of scanning the full time range through the cache path.

### Partition mode for aggregation

```json
{
    "query": {
        "sql": "SELECT kubernetes_namespace_name, COUNT(*) AS cnt FROM {stream} GROUP BY kubernetes_namespace_name ORDER BY cnt DESC LIMIT 10",
        "start_time": 1674000000000000,
        "end_time": 1675000000000000
    },
    "agent_options": {
        "mode": "partition"
    }
}
```

Each partition's aggregation result accumulates into a progressively merged cache entry. Follow-up queries with slightly shifted time windows reuse that cached work.

## Error Responses

When a search request fails, the API returns a standard error body. For certain error codes, the response includes **self-correcting guidance** to help you (or an AI agent) fix the query without manual investigation.

### Error Response Body

| Field name   | Data type     | Always present | Description |
|-------------|---------------|----------------|-------------|
| code         | int           | yes            | Numeric error code (see below) |
| message      | string        | yes            | Human-readable problem description |
| error_detail | string        | no             | Raw technical detail (omitted when `hint`/`suggestions` are present) |
| hint         | string        | no             | One-line guidance on how to fix the error |
| suggestions  | array[string] | no             | Closest valid alternatives (up to 3), ranked by similarity |

The `hint` and `suggestions` fields appear only when the server has enough information to offer useful guidance. Existing clients that ignore these optional fields are unaffected.

### Field Not Found (`code: 20004`)

When you reference a field that doesn't exist in the stream schema:

```json
{
  "code": 20004,
  "message": "unknown field 'servce'",
  "suggestions": ["service"]
}
```

The `suggestions` list contains the closest matching field names ranked by edit distance. If no close match is found, `hint` provides a fallback:

- **Small schemas** (10 fields or fewer): the `hint` lists all valid fields.
- **Large schemas**: the `hint` directs you to use the [stream schema endpoint](../stream/schema.md).
- **UDS violations** (the field exists in the raw stream but not in the User-Defined Schema): the `hint` explains that the field is excluded by the UDS and suggests adding it in stream settings.

### Function Not Defined (`code: 20005`)

When you call a function that doesn't exist:

```json
{
  "code": 20005,
  "message": "unknown function 'str_mach'",
  "hint": "usage: str_match(field, 'v')",
  "suggestions": ["str_match", "str_match_ignore_case"]
}
```

Suggestions cover both OpenObserve UDFs and DataFusion built-in functions. When a top suggestion is found, the `hint` carries a usage example for the closest match.

### Other Error Codes

| Code  | Name                          | Description |
|-------|-------------------------------|-------------|
| 20001 | `ServerInternalError`         | Internal server error |
| 20002 | `SearchSQLNotValid`           | SQL syntax is invalid |
| 20003 | `SearchStreamNotFound`        | Stream does not exist |
| 20004 | `SearchFieldNotFound`         | Field name not found (with suggestions) |
| 20005 | `SearchFunctionNotDefined`    | Function name not found (with suggestions) |
| 20008 | `SearchSQLExecuteError`       | SQL execution failed |
| 20009 | `SearchCancelQuery`           | Query was cancelled |
| 20010 | `SearchTimeout`               | Query timed out |
| 20011 | `InvalidParams`               | Invalid request parameters |
| 20012 | `RatelimitExceeded`           | Rate limit exceeded |
| 20013 | `SearchHistogramNotAvailable` | Histogram data unavailable |

## Next steps

- [Example queries](../../../user-guide/data-exploration/example-queries.md): copy-paste SQL examples to try.
- [Full-text search functions](../../sql-functions/full-text-search.md): `match_all`, `str_match`, `re_match`, and friends.
- [Logs UI](../../../user-guide/data-exploration/logs/logs.md): run these queries interactively.

**Need some help?**

- Join our [Community Slack](https://short.openobserve.ai/community) 
- Or [Contact support](https://openobserve.ai/contactus/)

---
title: Settings
description: Create or update OpenObserve stream settings via API. Configure partitions, index fields, full-text search keys, retention, and more.
---

## Set or Update Stream Settings

You can configure settings for a stream at creation time or update them later. Use the same endpoint with different HTTP methods depending on the operation.

## Create Stream Settings

Use this operation to define stream settings when creating a stream.

### Endpoint
```
POST /api/{org_id}/streams/{stream_name}/settings
```
### Request Body

Use the `StreamSettings` schema. All fields are optional.

```json
{
  "partition_keys": ["k8s_cluster", "k8s_namespace_name"],
  "index_fields": ["k8s_pod_name", "k8s_container_name"],
  "full_text_search_keys": ["body"],
  "bloom_filter_fields": ["k8s_node_name"],
  "data_retention": 30,
  "flatten_level": 1,
  "defined_schema_fields": [
    "body",
    "k8s_cluster",
    "k8s_pod_name",
    "k8s_app_component",
    "log_file_path",
    "service_name",
    "service_version",
    "severity"
  ],
  "max_query_range": 30,
  "store_original_data": true,
  "approx_partition": false,
  "extended_retention_days": [],
  "index_original_data": false,
  "index_all_values": false,
  "storage_type": "compliance"
}

```
### Response
```
{
  "code": 200
}
```

## Update Stream Settings
Use this operation to partially update an existing stream’s settings.

### Endpoint

```
PUT /api/{org_id}/streams/{stream_name}/settings
```
### Request Body

Use the `UpdateStreamSettings` schema. Fields that support `add`, `remove`, or `set` use wrapper syntax.

```json
{
  "partition_keys": {
    "set": ["k8s_cluster", "k8s_namespace_name"]
  },
  "full_text_search_keys": {
    "set": ["body"]
  },
  "index_fields": {
    "set": ["k8s_pod_name", "k8s_container_name"]
  },
  "bloom_filter_fields": {
    "set": ["k8s_node_name"]
  },
  "data_retention": 30,
  "flatten_level": 1,
  "defined_schema_fields": {
    "set": [
      "body",
      "k8s_cluster",
      "k8s_pod_name",
      "k8s_app_component",
      "log_file_path",
      "service_name",
      "service_version",
      "severity"
    ]
  },
  "max_query_range": 120,
  "store_original_data": true,
  "approx_partition": false,
  "extended_retention_days": {
    "add": []
  },
  "index_original_data": false,
  "index_all_values": false,
  "storage_type": "compliance"
}

```

### Response
```
{
  "code": 200
}
```

## Field Description
### StreamSettings Field Reference

| Field name                | Description                                                                                                                                                                       |
|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `partition_keys`          | Fields used to create data partitions, shown as `keyValue` or `Hash bucket` in the UI. Improves read performance by skipping unrelated files. Does not make the field searchable. |
| `index_fields`            | Fields to create secondary indexes for exact-match filters. Improves query performance on `field = value`.                                                                        |
| `full_text_search_keys`   | Fields tokenized for full-text search. Required for substring or `match_all` queries. Defaults to common log fields if not set.                                                   |
| `bloom_filter_fields`     | Fields with high-cardinality values to optimize rare value searches. Improves performance by skipping non-matching data blocks.                                                   |
| `data_retention`          | Number of days to retain data in the stream. Minimum is 3 days. Overrides the global retention setting. Compliance streams require a 30-day minimum.                              |
| `flatten_level`           | Maximum depth for flattening nested JSON objects into fields. Helps expose nested keys for querying.                                                                              |
| `defined_schema_fields`   | Fields to retain in the user-defined schema. Others are excluded or stored as raw if `store_original_data` is true.                                                               |
| `max_query_range`         | Maximum time range in hours for a single query. Prevents resource-heavy long-range queries.                                                                                       |
| `store_original_data`     | Stores the full original log body if schema filtering is applied. Allows retrieval of dropped fields.                                                                             |
| `approx_partition`        | Uses evenly divided time ranges for query execution. Helps distribute query load in skewed data.                                                                                  |
| `extended_retention_days` | List of time ranges to retain data beyond `data_retention`. Must be applied before data expires.                                                                                  |
| `index_original_data`     | Enables full-text indexing on the raw log body. Allows search across fields not part of the schema.                                                                               |
| `index_all_values`        | Indexes all fields for exact-match lookups. Increases ingestion and index size. Best for fixed schemas.                                                                           |
| `storage_type`            | Storage class for the stream: `normal` (default) or `compliance`. When `compliance`, compacted files are written to the infrequent-access storage class (requires `ZO_S3_FEATURE_FORCE_INFREQUENT_ACCESS=true`), and `data_retention` must be at least 30 days. |

---
title: Value
description: Search around to fetch field values from a stream using filters like time, keyword, and size with the _values API endpoint.
---

# Search around

Endpoint: `GET /api/{organization}/{stream}/_values?fields={fields}&start_time={start_time}&end_time={end_time}&size=10&keyword=&no_count=false`

## Request

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| stream     | string    | -             | stream name |
| fields     | string    | -             | the fields you want to get values, `field1,field2` |
| size       | int64     | 0             | how many values do you want to response, order by values num |
| start_time | int64     | 0             | Only list the values in the time range |
| end_time   | int64     | 0             | Only list the values in the time range |
| keyword    | string    | -             | search for the values |
| no_count   | bool      | false         | set to `true` will not response count and order by the value |

## Response

```json
{
	"took": 155,
	"hits": [
		{
			"field": "field name",
			"values": [
				{
                    "zo_sql_key": "value1",
                    "zo_sql_num": 2070
                }
			]
		}
	],
	"total": 10,
	"from": 0,
	"size": 0,
	"scan_size": 28943
}
```

Response description:

Description

| Field name | Data type | Default value | Description |
|------------|-----------|---------------|-------------|
| took       | int64     | -             | unit: milliseconds, query execute time |
| from       | int64     | 0             | value from `query.from` |
| size       | int64     | 0             | value from `query.size` |
| scan_size  | int64     | 0             | unit: MB, it response the data size scale when execute the query. |
| hits       | array     | -             | records for query, each record is a log row what you ingested. |