Control: input paths & semantic (element) control

How taps, swipes, typing, and key presses reach the device — and how to target UI elements instead of raw pixels.

Input paths

The server picks the injection path automatically per capture backend:

• emulator gRPC (--capture grpc) — sendTouch / sendKey RPCs. Coordinates are device pixels; nav keys use W3C DOM names (GoHome, GoBack, AppSwitch).
• scrcpy control socket (scrcpy mode) — control messages encoded to scrcpy's v4.1 binary protocol and written straight to its control socket (no per-event process spawn), so latency is low. Taps, swipes (synthesized DOWN→MOVE→UP), text, and nav keys. Disable with --scrcpy-control off.
• adb input — screenrecord mode, or scrcpy with control off. Universal but spawns an adb process per event.

The scrcpy backend opens a second connection to the forwarded port for the control socket. scrcpy accepts video first, then control, and blocks video until the control socket connects — so stream-droid opens control as soon as the video socket is established (see src/capture/scrcpy.ts).

Normalized coordinates. The browser sends [0..1] positions; the server scales them to device pixels from adb shell wm size per device, so taps stay correct across resolutions and rotation (and the scrcpy backend needs no size header).

Typing caveat. adb input text escapes spaces as %s and is fussy with punctuation; swap in an ADB IME (e.g. ADBKeyBoard) for reliable full-text entry.

Clipboard (paste & copy). The browser supports ⌘/Ctrl+V to paste and ⌘/Ctrl+C to copy the device's clipboard:

• paste works on all three backends: scrcpy's SET_CLIPBOARD (populates the device clipboard for on-device reuse, handles multiline/non-ASCII); gRPC's text RPC (unicode-safe, no device clipboard); adb input text split on newlines with KEYCODE_ENTER (inherits adb input text punctuation quirks).
• copy is scrcpy-only; there is no reliable way to read the clipboard over plain adb, and the bundled gRPC proto has no clipboard RPC. On the default screenrecord backend, ⌘C behaves normally (no clipboard copy). The copy request uses scrcpy's COPY_KEY_COPY default, which also injects KEYCODE_COPY into the device first — so a host-side ⌘C copies a live on-device text selection into the device clipboard (and then reads it back), mutating the device clipboard as a side effect even if the client already had a cached value.

Semantic control (elements, not pixels)

All three backends are pixel-based; for agents that need to target UI elements and read on-screen state, stream-droid exposes the accessibility/view hierarchy over plain adb — uiautomator dump, no Appium server required. Works on emulators and physical devices (src/semantic.ts).

• GET /api/hierarchy?serial=… → JSON of every node: resourceId, text, desc, className, clickable, bounds, and center.
• { "type": "tapElement", "id": "…" | "text": "…" } over the WebSocket — the server resolves the first matching element's center and taps it via the active input backend. id matches the full resource-id or its short tail; text matches exactly or as a substring (of text or content-desc).

This is robust against resolution/layout changes where fixed coordinates would break. For richer driver semantics (smart waits, complex gestures, cross-platform scripts), Appium / UiAutomator2 remains the heavier option.

Clipboard notifications (server → client)

The server sends these WebSocket messages to keep the client's clipboard cache current:

• { "type": "meta", "clipboard": boolean, … } — advertises whether copy can work on the current backend. Sent on connect.
• { "type": "clipboard", "value": "…" } — pushed whenever the device clipboard changes (scrcpy's clipboard_autosync). Sent only to control-authorized clients — a view-only viewer over a shared tunnel never receives the device clipboard, as it may contain sensitive data (passwords, etc.).

Programmatic API

The HTTP routes (/api/state, /api/start, /api/hierarchy) and the WebSocket frame + control protocol are documented, agent-focused, in the skill references: http-api.md and websocket.md.