KafkaLoader connected to Confluent Cloud

Hi, im trying to connect kafka loader to confluent cloud. The following config feels close, I don’t appear to get any errors coming out other than a timeout on “Try to list topic metadata from Kafka broker”

It also seems that none of the log files are being written to on /home/tigergraph/tigergraph/log) - i’m on the docker container: docker.tigergraph.com/tigergraph:latest

    {
    	"broker": "SERVER_ID.eu-west-1.aws.confluent.cloud:9092",
    	"kafka_config": {
    		"bootstrap.servers": "SERVER_ID.eu-west-1.aws.confluent.cloud:9092",
    		"group.id": "tigergraph",
    		"security.protocol":"plaintext",
    		"sasl.mechanism":"PLAIN",
    		"sasl.username":"API_KEY",
    		"sasl.password":"API_SECRET",
    		"schema.registry.url":"https://AAA.eu-central-1.aws.confluent.cloud",
    		"basic.auth.credentials.source":"USER_INFO",
    		"basic.auth.user.info":"SR_API_KEY:SR_API_SECRET"
    	}
    }

New findings - providing “sasl.jaas.config” means it doesn’t timeout or give the error “Failed to construct kafka consumer”

However, it does give the error: Java JAAS configuration is not supported. Latest config:

    {
	"broker": "XYD.eu-west-1.aws.confluent.cloud:9092",
	"kafka_config": {
		"bootstrap.servers": "XYD.eu-west-1.aws.confluent.cloud:9092",
		"group.id": "tigergraph",
		"sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule required username=\"XXX\" password=\"UUU\";",
		"security.protocol":"SASL_SSL",
		"sasl.mechanism":"PLAIN",
		"ssl.ca.location":"/etc/ssl/certs/",
		"sasl.username":"XXX",
		"sasl.password":"YYY"
	}
}

i’ve tested these parameters using kafkacat as detailed here:

Albeit that this one requires sasl.mechanisms instead of sasl.mechanism - with kafkacat the same config works and consumes from the topics.

The obvious thing to do is to drop sasl.jaas.config but then it doesn’t get past (i assume) some kinda validator

For the next person finding to this - I believe my issues have been coming from the tigergraph docker container (both official and xpertmind)
https://github.com/experoinc/graph-based-rbac-with-streaming-synchronization (i think this one has issues with java libraries, causing the sasl error)
https://github.com/xpertmind/TigerGraph (my guess is a networking error on this one now, but could also be missing packages to work with the remote kafka stream, e.g ssl bundles had to be installed initially, so it’s obv really minimal)

The following config passes authentication on xpertmind:

    {
    	"broker": "XYD.eu-west-1.aws.confluent.cloud:9092",
    	"kafka_config": {
    		"bootstrap.servers": "XYD.eu-west-1.aws.confluent.cloud:9092",
    		"group.id": "tigergraph",
    		"sasl.jaas.config":"org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='UUU';",
    		"security.protocol":"SASL_SSL",
    		"sasl.mechanism":"PLAIN"
    	}
    }

It then stalls at trying to read the data out of confluent cloud - but i’m starting to think that may be something specific to the docker-compose. I’m currently awaiting GSQL access on Tigergraph Cloud without credit card which should just allow this to work with no further complications

@thepauleh what version of TG Docker are you using? I’m maintaining xpertmind Docker image, so maybe we can find what’s wrong there. What is your docker-compose looking like ?

Bruno

Hi @Bruno i’m on xpertmind/tigergraph:latest (downloaded today). I discovered I can only seem to have one kafka-loader running at a time - which is why it appeared to ‘stall’ reading the data after passing authentication.

For the more reliable results i’m restarting the container in between each loader installation to prevent that confusion around the loader state.

This then leads me back to my second post with the complication around ‘sasl.jaas.config’ giving the following error if supplied:
failed with error: Java JAAS configuration is not supported, see https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka for more information

and the other error if not supplied:
Failed to construct kafka consumer

OK, I just changed something in the image and am pushing the new version online (latest / 3.3.0)
Please try with it. (docker-compose pull && docker-compose up -d)
I also published an updated docker-compose.yaml file.

Bruno