Backend and sidecar

The executable part of a package never runs inside the application’s process like a library. It runs in a runtime of its own, is handed its bytes by the host, and reaches the outside world only through a mediated surface.

Two kinds, one manifest field:

// in-process: one JavaScript module in an isolated runtime
"backend": { "entry": "backend.js" }

// sidecar: your own process, your own language, your own libraries
"backend": {
  "entry": "runtime/main.py",
  "runtime": "sidecar",
  "requires": { "language": "python", "minVersion": "3.11" }
}

Take in-process unless something argues against it. It is smaller, it starts faster, and it runs inside a real sandbox.

Take a sidecar when you need to hold a connection open, or when the library you need does not exist in JavaScript. That is exactly what industrial connectors are: a socket that stays open and reports when something arrives — not a function that answers a question.

The price of a sidecar is honest and worth reading twice: it is a real process on the operator’s machine. The host starts it, watches it and takes it down — it does not sandbox it.

---

The in-process backend

Your entry module exports one function that receives the mediated host surface and returns an object with routes and handle.

export default (host) => {
  async function query(sql, params) {
    const res = await host.request({ method: 'POST', path: 'data/query', body: { sql, params } });
    return res && res.body && Array.isArray(res.body.rows) ? res.body.rows : [];
  }

  return {
    routes: [
      { method: 'GET', path: '/reports' },
      { method: 'POST', path: '/reports' }
    ],

    handle: async (env) => {
      const segments = String(env.path || '').split('/').filter((s) => s.length > 0);
      const caller = env.caller;

      // THE GATE. Server-side, before the route is even chosen. An absent caller is
      // "nobody proved anything" and is refused, never treated as trusted.
      const roles = caller && Array.isArray(caller.roles) ? caller.roles : [];
      if (env.method !== 'GET' && !roles.includes('tenant-admin')) {
        return { status: 403, body: { error: 'not_allowed_to_write' } };
      }

      if (segments[0] === 'reports' && env.method === 'GET') {
        return { status: 200, body: { rows: await query('SELECT * FROM plugin_pkg.acme_reports') } };
      }
      return { status: 404, body: { error: 'no_such_route' } };
    }
  };
};

Two rules follow from this:

• routes is reported at start. The host knows the routing table before the first request arrives. What is not declared is not delivered.
• handle must always answer. An unhandled throw becomes a 500 without killing the worker — but your caller then waits for the timeout. Catch, and return a code; the message stays internal.

Your route is reachable at /api/plugins/<pluginId>/<path>.

The envelope

What handle receives, and what it must return:

// in
{ method, path, headers?, body?, caller? }
// out
{ status, headers?, body? }

env.caller is stamped by the host from the session: { appUserId, roles, kind }. A caller key inside env.body is the caller’s own text and stays there — it can never reach this field. A backend that states a caller of its own on the reverse channel is refused with caller_not_settable.

caller is optional: a background call has none. Read “no caller” as “nobody proved anything” — fail-closed, never as trusted.

The mediated host surface

host.request({ method, path, body }) is the only way out. The paths are a closed set:

Path	What for	Needs
data/query	One parameterised query, tenant-bound	data:read / data:write
data/batch	Several statements, atomic within one invocation	as above
audit	An audit line. The actor is set host-side, never by you	—
events	A real-time event through the mediated bus	—
log	A line into the central log	—
net/request	One outbound HTTP(S) request, mediated by the host	net:request
capability/invoke	Invoke a host capability, e.g. tag.write	capability:invoke plus the capability’s own permission
host/resolve	A business name → a core id (tag, recordType, field, role)	—
host/describe	The reverse: an id you already own → its name	—

The last two have sugar that throws on failure instead of returning a non-200 answer:

const tagId = await host.resolve('tag', 'Press1.PartCount');
const tagPath = await host.describe('tag', tagId);

host/describe is not a browsing hole: an id must already be yours — sitting in your own row — to be described at all. Describing is never how you find an id you did not already have.

What data/query may reach is not decided by a text filter over your SQL, but by the role it runs under: it cannot leave the tenant-scoped area. There is deliberately no denylist of statements.

Limits the host sets

Limit	Value
Backend start	10 s
One invocation	30 s
Drain on cutover	30 s
Restart attempts after a crash	3, waiting 1 s / 5 s / 30 s

---

The sidecar

A sidecar is a process of its own. The host starts it, checks the interpreter first, and speaks to it over three channels:

fd	Direction	Content
0 (stdin)	Host → sidecar	framed frames
1 (stdout)	Sidecar → host	framed frames
2 (stderr)	Sidecar → host	free text, line by line, never framed

stderr stays unframed because a traceback at start happens before the framing exists. The host logs every line as your plugin’s log at level warn.

Your sidecar must use stdout for nothing else. A print() destroys the stream. Redirect sys.stdout to sys.stderr at start and keep the real descriptor to yourself.

The framing

+--------+------------------------+
| 4 byte | N byte                 |
| length | UTF-8 JSON             |
+--------+------------------------+

Length is big endian, unsigned, and does not count itself. The cap is 8 MiB per frame. UTF-8 is decoded strictly — a mangled byte must not quietly become U+FFFD, or it lands in a tag looking like a healthy value.

Error names, so both halves log the same thing: frame_too_large, invalid_frame_length (length 0), incomplete_frame, invalid_frame_json, unknown_frame.

After the first framing error the reader is poisoned: it never re-aligns, it throws the same error on every further byte. A stream whose framing has slipped cannot be recovered; every attempt would be guessing.

The sequence

Host                                Sidecar
 |-- spawn(python, entry) ---------->|
 |-- load -------------------------> |
 |                                   |  digest over the tree
 |  <------------------------ ready -|
 |-- request (POST /connections) --> |
 |  <--------------------- response -|   socket opens
 |  <------------------------ event -|   telegram received
 |  <------------------------- host -|   capability/invoke tag.write
 |-- host-response ----------------> |
 |-- shutdown ---------------------> |

The first frame is load and names rootPath, entry, archiveDigest and the limits. rootPath points at the already unpacked and verified package — your sidecar unpacks nothing and verifies no signature.

Your answer is ready with routes, archiveDigest and runtimeVersion. You compute the digest yourself over the tree, and the host compares it with the one from load. A difference is entry_digest_mismatch, and the process is terminated.

How the tree digest is formed, so both sides compute the same thing: all files relative to rootPath, paths with /, sorted by their UTF-8 bytes; per file sha256(path-as-utf8) || sha256(content) concatenated as raw 32 bytes; sha256 again over the whole. Directories do not count. The result carries the prefix sha256:.

Frames in operation

• request / response — paired by correlationId; envelope is the same envelope as above.
• host / host-response — your reverse channel. The same operations as the in-process backend, plus capability/invoke.
• event — sidecar → host, unsolicited, with no correlationId. This is the frame the whole design exists for: a held socket reports when something arrives, not when somebody asks.
• shutdown — the polite request. It is enforced by closing stdin, then a grace period, then killing.

Your sidecar needs two threads

Not a design detail but an inevitability: a host call waits for a frame only the read loop can read. Calling the host from inside the read loop deadlocks you against yourself.

So: the read loop only reads. Incoming request frames go to a worker thread; host-response is resolved in the read loop. A host call from the reading thread is refused — with an error, not with a deadlock.

The socket is yours

There is no net/listen, net/connect or net/send. A host-mediated socket would look safer. It would not be: a Python process has ambient network authority and could bypass the mediation anyway. We would have built an enforcement that enforces nothing, and paid for every received byte twice through the framing.

Hence the split: data (the socket) is held by the sidecar itself; control (tags, logs, audit, database) is host-mediated, because there the enforcement is real — the sidecar has no other route there.

net:socket in the manifest is therefore a declaration, not a cage: it is checked, logged and audited, and a package without it is never started as a connector in the first place. It does not stop a malicious sidecar. Only an OS-level boundary could, and there is none here.

What a sidecar never gets

• No database connection and no credentials for one.
• No environment variables beyond the allowlist.
• No open file descriptors except 0, 1, 2.
• No write access outside its own working directory.

---

A connector sidecar

A connector plugin is a package with a connector contribution and a sidecar backend. The type alone is a form with no effect; the sidecar alone is a process with no configuration.

It serves five routes, and no more:

Route	What it does
POST /connections	Create or change a connection. { id, settings }
GET /connections	What the sidecar is actually holding right now
GET /connections/:id	One connection’s state; unknown → 404
DELETE /connections/:id	Close and forget. Twice is not an error
POST /connections/:id/send	Send outward. { payload } → { sent: n }

Called again with the same id means CHANGE, not open a second time. The host calls this after every change and after every sidecar restart. A sidecar that opens a second socket here ends up holding as many as the operator has saved.

sent: 0 is an ERROR, not a success. Whoever writes outward gets a receipt. A server with no connected peer sent nothing, and a tag that then shows “written” is lying to the operator.

And it sends two events:

{ "kind": "event", "type": "connector/received",
  "payload": { "connectionId": "c1", "payload": "OK", "bytes": 2,
               "terminator": "delimiter", "peer": "10.0.0.5:51234" } }

terminator is delimiter | silence | eof. With silence the telegram was not ended by a control character but because the line went quiet — a cut at a possibly wrong place, visible only if it is reported.

{ "kind": "event", "type": "connector/state",
  "payload": { "connectionId": "c1", "connected": true, "lastError": "",
               "rxCount": 17, "txCount": 3 } }

The keys of this payload are the channel names. The host writes each of them into the tag that chose that channel, and does not know the names itself. A connector with a fifth state value declares it in its channel field, sends it here, and the host records it — without a line of core code.

Permissions for a connector

Permission	What for
net:socket	The held socket. Without it this is not a connector.
capability:invoke	Opens the reverse channel.
data:write	Sits behind it and is easily forgotten: tag.write is mounted under this permission. Without it every received telegram ends as permission_denied instead of a tag value.

data:read only if your connector really reads tags.

When writing into a tag, set inbound: true when the value came from the peer:

{ "op": "capability/invoke",
  "payload": { "name": "tag.write",
               "input": { "path": "Line1.Rx", "value": "OK", "inbound": true } } }

Without this field a two-way tag sends the telegram it just received straight back out. It is origin, not authority: the only effect is to skip the outbound send. Every gate stays where it was. The tenant is set by the host from the invocation context, never from your payload.

What the host does so you do not rebuild it

• It keeps the sidecar alive and restarts it after a crash.
• It calls POST /connections for every saved connection after every start.
• It turns connector/received and connector/state into tag writes — under the actor machine, with an audit entry.
• It turns a write to an outbound tag into a POST /connections/:id/send.
• It refuses a write to a state tag. connected is a measurement, not a switch.

What is open today

• One sidecar per package. Several processes out of one package are not forbidden, just not thought through.
• One connector type per package. Likewise.
• No request/response pairing on the socket. Send-and-wait-for-the-answer does not exist as a route; two tags and a script trigger rebuild it.
• No bundled dependencies. The interpreter is a precondition of the installation, not package content.
• No throughput promise. At a high telegram rate the channel’s framing is the first place that hurts: every telegram is a frame.