Data Pipe

WARNING

Manual editing is usually not required; you typically only need to fill in the main variables because there is a file that generates .env files for services.

Variable	Default	Purpose
PU_DP_DOMAIN	-	Domain name or IP – used to correctly subscribe to topics
PU_DP_SECRET_KEY	-	32‑byte key in base64 format. Used to sign auth tokens. If changed, all previously issued jwt tokens become invalid
PU_DP_SQLALCHEMY_DATABASE_URL	-	Connection URL for PostgreSQL
PU_DP_CLICKHOUSE_DATABASE_URL	-	Connection URL for ClickHouse
PU_DP_REDIS_URL	redis://redis:6379/0	Redis connection URL used to exchange UnitNode configurations between Backend and DataPipe
PU_DP_MQTT_HOST	-	Domain name or IP used to connect to EMQX
PU_DP_MQTT_PORT	1883	Port used to connect to EMQX
PU_DP_MQTT_KEEPALIVE	60	Maximum period in seconds between ping messages from DataPipe to EMQX
PU_DP_CONFIG_SYNC_INTERVAL	60	Interval in seconds between repeated YML configuration requests
PU_DP_NRECORDS_CLEANUP_INTERVAL	60	Interval in seconds between cleanup runs that remove excess N Records data
PU_DP_BUFFER_FLUSH_INTERVAL	5	Interval in seconds after which data will be flushed to ClickHouse or PostgreSQL regardless of the number of accumulated records
PU_DP_BUFFER_MAX_SIZE	1000	Maximum queue length; when exceeded, data is written to ClickHouse or PostgreSQL immediately without waiting for PU_DP_BUFFER_FLUSH_INTERVAL. The same queue and the same interval are used for DataPipe alerts
PU_FF_NOTIFICATION_ENABLE	True	True / False. Together with the same flag on the Backend, enables writing pipeline alerts to notifications. When the flag is off, the alerts block in the pipeline does not create records in PostgreSQL