Stream Events to Your Own Broker
Publish workflow records to Pub/Sub, Kafka, SNS, SQS or AMQP — your topics, your infrastructure, exactly-once semantics.
For teams that want events, not exports: the message-queue action publishes every workflow record to a broker you own — Google Pub/Sub, Kafka, AWS SNS, SQS, or any AMQP broker. Auth is grown-up: for Pub/Sub, either grant a dedicated low-privilege publisher service account access to your topic (no secret stored) or supply a scoped key that's encrypted at rest. Delivery is idempotency-wrapped, supports per-record ordering keys, and there's a connectivity probe before you rely on it.
Workflow triggers
Inbound: use webhooks or schedules
There is no broker-consume trigger — inbound paths are HTTP webhooks, polling reads and hot folders. This page is about getting events OUT reliably.
Workflow actions
Publish to Pub/Sub, Kafka, SNS, SQS, AMQP
One action, five broker adapters — pick the connection, records flow as messages.
Ordering & idempotency
Per-record ordering keys where the broker supports them; dispatch-wrapped idempotency means redeliveries replay the cached result instead of double-publishing.
Connectivity test
A delivery-test endpoint verifies the connection before production traffic.
How data flows
Outbound only, by design — the durable alternative to webhook callbacks. Survey responses, enriched records and workflow outputs land on your topics; your consumers do the rest.
Connection: Per-broker connections created once, with the destination topic fixed at creation. Pub/Sub supports IAM grant (no stored secret) or an encrypted scoped service-account key; other brokers use their native credentials.
Worked example: Responses into your data platform, event-style
Create a Pub/Sub connection: grant the publisher service account roles/pubsub.publisher on your topic — no keys exchanged.
Trigger: every survey response runs an enrichment workflow.
Action: the enriched record publishes to your topic with the respondent id as ordering key.
Your Dataflow/consumer pipeline picks it up like any other event source.
Setting it up
1.
Create a broker connection (Pub/Sub IAM grant or scoped key; native credentials for Kafka/SNS/SQS/AMQP).
2.
Run the connectivity test.
3.
Add the message-queue action to any workflow and select the connection.
Frequently asked questions
Which brokers are supported?
Google Pub/Sub, Kafka, AWS SNS, AWS SQS, and AMQP-compatible brokers — one action, five adapters.
How is Pub/Sub authenticated?
Two modes: grant our dedicated low-privilege publisher service account roles/pubsub.publisher on your topic (no secret ever stored), or supply a scoped service-account key, encrypted at rest.
Can SurveyAnalytica consume from my broker too?
Not today — inbound is via webhooks, scheduled polling reads and hot folders. Streaming is outbound.