Datastore API

DRAFT — Internal Developer Use Only

This API reference is for internal development teams.

Overview

What it is: The Datastore API provides direct access to Hydden's entity and edge storage layer. It supports retrieving collection metadata, querying entities with filtering and history, creating/updating/deleting entities and edges, and managing the datastore module configuration. For backup/restore operations, see the Backup & Restore API.

Source: src/datastore/rest/rest.go

Base Path

/internal/v1/datastore

Authentication

All endpoints require JWT cookie or API token authentication.

Endpoints

Method	Path	Description
GET	/internal/v1/datastore/info	Get collection store information
GET	/internal/v1/datastore/shard/info	Get shard replication information
POST	/internal/v1/datastore/fetch	Fetch entities with filtering and transformation
GET	/internal/v1/datastore/entity/:target/:id	Get a single entity
PUT	/internal/v1/datastore/entity/:target/:id	Create or update an entity
DELETE	/internal/v1/datastore/entity/:target/:id	Delete an entity
PUT	/internal/v1/datastore/edge/:target/:from/:to	Create or update an edge (relationship)
GET	/internal/v1/datastore/module/config	Get datastore module configuration
POST	/internal/v1/datastore/module/config	Update datastore module configuration

---

GET /internal/v1/datastore/info

Get information about collection data stores, including view times, entity counts, and replication node details.

Request:

GET /internal/v1/datastore/info
Authorization: Bearer <token>

Response (200):

{
  "viewTime": 1707700800000,
  "maxTime": 1707700800000,
  "stores": [
    {
      "id": "ds-uuid",
      "viewTime": 1707700800000,
      "count": 4521,
      "nodes": [
        { "id": "node-1", "name": "Primary", "cloud": false, "online": true, "count": 4521 }
      ],
      "tombstoned": false,
      "name": "Corporate AD",
      "platform": "Active Directory",
      "version": 42
    }
  ],
  "historical": false,
  "node": "node-1",
  "cached": false,
  "version": 42
}

Response fields (CollectionInfo):

Field	Type	Description
viewTime	int64	Current view timestamp (ms)
maxTime	int64	Maximum available timestamp
stores	array	Collection store entries
stores[].id	string	Data source ID
stores[].count	int64	Number of entities in this store
stores[].name	string	Data source display name
stores[].platform	string	Platform type (Active Directory, Azure AD, etc.)
stores[].nodes	array	Replication nodes holding this store
stores[].tombstoned	bool	Whether the store is soft-deleted
historical	bool	Whether viewing historical data
node	string	Responding node ID
cached	bool	Whether the response is from cache

---

GET /internal/v1/datastore/shard/info

Get shard distribution and replication information across all nodes.

Request:

GET /internal/v1/datastore/shard/info
Authorization: Bearer <token>

Response (200):

{
  "replicas": 2,
  "stores": 12,
  "total": 45230,
  "nodes": [
    {
      "id": "node-1",
      "name": "Primary",
      "cloud": false,
      "online": true,
      "count": 45230,
      "stores": [],
      "version": 42
    }
  ],
  "node": "node-1",
  "cached": false
}

Response fields (ReplicationInfo):

Field	Type	Description
replicas	int64	Configured replication factor
stores	int64	Total number of collection stores
total	int64	Total entity count across all stores
nodes	array	Replication nodes with status and entity counts
nodes[].online	bool	Whether the node is currently reachable
nodes[].cloud	bool	Whether this is a cloud node

---

POST /internal/v1/datastore/fetch

Fetch entities from a collection store with optional filtering and transformation. Results are streamed as a JSON array.

Request:

POST /internal/v1/datastore/fetch
Content-Type: application/json
Authorization: Bearer <token>

{
  "dataSourceId": "ds-uuid",
  "entity": {
    "entityType": "identity.user",
    "time": 1707700800000
  }
}

Query parameters:

Parameter	Type	Description
limit	int	Maximum entities to return
filter	string	Filter expression applied to entities
transform	string	Transform function applied to results

Response (200): Streaming JSON array of entities (chunked transfer encoding).

[
  {
    "uniqueId": "acct-uuid-1",
    "entityType": "identity.user",
    "displayName": "John Doe",
    "email": "john.doe@company.com"
  }
]

---

GET /internal/v1/datastore/entity/:target/:id

Get a single entity by data source ID and entity ID, with optional history.

Request:

GET /internal/v1/datastore/entity/:target/:id
Authorization: Bearer <token>

Path parameters:

Parameter	Description
target	Data source ID
id	Entity ID (base64 encoded)

Query parameters:

Parameter	Type	Description
history	bool	Include entity change history (default: false)
layer	int[]	Specific data layers to fetch (repeatable)

Response (200):

{
  "id": "acct-uuid",
  "dataSourceId": "ds-uuid",
  "type": "identity.user",
  "entity": {
    "displayName": "John Doe",
    "email": "john.doe@company.com",
    "upn": "jdoe@corp.local"
  },
  "time": 1707700800000,
  "tombstoned": false,
  "version": 5,
  "storeVersion": 42,
  "layers": [1, 2],
  "history": [
    {
      "time": 1707614400000,
      "tombstoned": false,
      "entity": {},
      "layers": [1]
    }
  ]
}

Response fields (TargetEntity):

Field	Type	Description
id	string	Entity unique ID
dataSourceId	string	Data source ID
type	string	Entity type identifier
entity	object	Deserialized entity data
time	int64	Last modification timestamp (ms)
tombstoned	bool	Whether the entity is soft-deleted
version	uint64	Entity version number
storeVersion	uint64	Collection store version
layers	int64[]	Data layers present for this entity
history	array	Historical versions (when ?history=true)

---

PUT /internal/v1/datastore/entity/:target/:id

Create or update an entity in the datastore.

Request:

PUT /internal/v1/datastore/entity/:target/:id?type=identity.user
Content-Type: application/json
Authorization: Bearer <token>

Path parameters:

Parameter	Description
target	Data source ID
id	Entity ID

Query parameters:

Parameter	Type	Required	Description
type	string	Yes	Entity type (e.g., identity.user, identity.group)

Request body: JSON entity matching the protobuf schema for the specified type.

{
  "displayName": "John Doe",
  "email": "john.doe@company.com"
}

Response (200): JSON echo of the stored entity.

---

DELETE /internal/v1/datastore/entity/:target/:id

Delete an entity from the datastore (creates a tombstone marker).

Request:

DELETE /internal/v1/datastore/entity/:target/:id
Authorization: Bearer <token>

Response (200): Empty response on success.

---

PUT /internal/v1/datastore/edge/:target/:from/:to

Create or update a relationship edge between two entities.

Request:

PUT /internal/v1/datastore/edge/:target/:from/:to?type=edge.identity.manager
Content-Type: application/json
Authorization: Bearer <token>

Path parameters:

Parameter	Description
target	Data source ID
from	Source entity ID
to	Target entity ID

Query parameters:

Parameter	Type	Required	Description
type	string	Yes	Edge type (must start with edge., e.g., edge.identity.manager, edge.identity.account)

Request body: JSON edge properties.

Response (200): JSON edge with computed unique ID.

---

GET /internal/v1/datastore/module/config

Get the datastore module configuration, including replication settings and node assignments.

Request:

GET /internal/v1/datastore/module/config
Authorization: Bearer <token>

Response (200):

{
  "id": "datastore-module",
  "name": "Datastore",
  "replicas": 2,
  "weights": {
    "node-1": 100,
    "node-2": 100
  },
  "nodes": [
    { "id": "node-1", "name": "Primary", "tenant": "tenant-uuid" },
    { "id": "node-2", "name": "Secondary", "tenant": "tenant-uuid" }
  ],
  "version": 5,
  "site": "us-east-1",
  "refresh": false
}

Response fields (ModuleConfig):

Field	Type	Description
id	string	Module configuration ID
name	string	Module display name
replicas	int64	Replication factor
weights	map	Node-to-weight mapping for data distribution
nodes	array	Assigned nodes
version	uint64	Configuration version
site	string	Deployment site identifier
refresh	bool	Whether a refresh is pending

---

POST /internal/v1/datastore/module/config

Update the datastore module configuration.

Request:

POST /internal/v1/datastore/module/config
Content-Type: application/json
Authorization: Bearer <token>

Request body uses the same ModuleConfig structure. The id field is required.

Response (200): Updated ModuleConfig.

---

Error Responses

Status	Description
400	Invalid request body, missing type parameter, or invalid entity ID encoding
403	Authentication failed or insufficient permissions
404	Entity or data source not found
500	Internal server error

Related Topics

• Backup & Restore API — Datastore backup and restore operations
• Entity Management API — Entity index operations
• Registry Configuration API — System configuration storage