Kafka Transmit

(Deprecated: RDPv3 is the preferred transmit method for transporting data to kafka.) Transfers data to the Sight Machine platform Kafka Instances.

The following configuration excerpt uploads data to the associated Sight Machine cloud instance:

{
    "transmit_name": "SightMachine",
    "transmit_type": "kafkaproducer",
    "filter_stream": ["*"]
}

For example, this configuration snippet uploads data to factorytx-dev.sightmachine.io:

{
    "sight_machine_cloud": {
      "base_url": "https://factorytx-dev.sightmachine.io",
      "api_key_id": "factory_5a6a52dce367100011872587@sightmachine_ftx.com",
      "api_key": "SeyYnA1OUL2fw6rFbh5A43RvTPDduoQqYaPDOWqofeR",
    },
    "data_transmit": [
        {
            "transmit_name": "SightMachine",
            "transmit_type": "kafkaproducer",
            "username": "username",
            "password": "supersecretpassword",
            "tenant": "tenant-name",
            "broker_hosts": "kafka.sm.io:9093,kafka2.sm.io:9093,kafka3.sm.io:9093"
        }
    ]
}

Configuration:

Required and optional properties that can be configured for a Kafka transmit.

  • transmit_name: ID for the transmit. It must be unique.

  • transmit_type: Method to use in transmitting records. The default type is kafkaproducer.

  • broker_hosts: List of comma-separated server:port, such as kafka.sm.io:9093,kafka2.sm.io:9093

  • tenant: Base name of the topic(s) to produce Kafka messages to.

  • filter_stream: A list of streams that will use the transmit. Each stream can either be * (all) or asset:stream.

  • compression: Compression codec that the Kafka producer uses to compress message sets to the server. Supported types of compression are: gzip, snappy, lz4, and zstd. The default compression type is snappy.

  • timeout: Number of seconds to wait until timing out.

  • poll_interval: Maximum number of seconds to wait between kafka requests.

  • max_request_records: Maximum number of records to send to Kafka before waiting for a response.

  • max_request_size_bytes: Maximum number of bytes allowed in a single request. Regardless of this size, it is capped at 4 MB.

  • use_ssl: Use SSL and SASL authentication

  • username: Username for authenticating against a Kafka Cluster - required if use_ssl is true

  • password: Password for authenticating against a Kafka Cluster - required if use_ssl is true

  • ssl_endpoint_identification_algorithm: Endpoint identification algorithm to validate broker hostname using broker certificate. - defaults to https.

  • enable_idempotence: If enabled, FactoryTX will use a Kafka Idempotent Producer for exactly-once producing. Please note that the configured user will need to have ‘IdempotentWrite’ privileges. The default is false.

  • topic_field: Name of the field present in dataframe to get topic suffix. If topic_field not specified then default topic suffix is stream id - Optional