OCI GoldenGate Service to Kafka Use case: Step-by-step guide for replication of data to Kafka using OCI GoldenGate Service BigData. The scope of this Blog is limited to Replicat information.
This Architecture will help in replicating the data in real-time to any of the cloud managed Kafka/Custom installed Kafka on-premise. With the introduction of full managed service by OCI, GoldenGate can be used to send the data into Kafka Topics. This PaaS service is completely based on OCPU consumption and no more requirement of licensing the Source and Target systems.
Prerequisites :
STEP 1 : Create Deployment in OCI
STEP 2 : Create the connection
Note : Please select the security protocol based on target Kafka environment. Most of the production systems will be with SASL.
Choose accordingly.
PRIVATE IP : Please put the private IP address of Target Kafka.
STEP 3 : Assign the connection to Deployment.
LAUNCH CONSOLE to create the replicat process.
Fill in the details about target, trail name and the connection information
Make sure to select Target as Kafka & select the right available aliases
Change the Map statement based on source & target or can be left with *.*
Sample props file
# Properties file for Replicat RKAFKA
#Kafka Connect Handler Template
gg.handlerlist=kafkaconnect
gg.handler.kafkaconnect.type=kafkaconnect
gg.handler.kafkaconnect.connectionId=ocid1.goldengateconnection.oc1.iad.ama
gg.handler.kafkaconnect.mode=op
#TODO: Set the template for resolving the topic name.
gg.handler.kafkaconnect.topicMappingTemplate=${fullyQualifiedTableName}
gg.handler.kafkaconnect.keyMappingTemplate=${primaryKeys}
gg.handler.kafkahandler.SchemaTopicName=OGGBD
gg.handler.kafkaconnect.messageFormatting=row
gg.handler.kafkaconnect.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}
gg.handler.kafkaconnect.converter=json
#TODO: Set the location of the Kafka client libraries
gg.classpath=$THIRD_PARTY_DIR/kafka/*
jvm.bootoptions=-Xmx512m -Xms32m
Check the statistics & the topics for the number of records.
VALIDATE THE TARGET KAFKA BY RUNNING CUSTOM COMMANDS
kafka-run-class.sh kafka.tools.GetOffsetShell — broker-list ogg21cbigdata:9092 — topic ADDRESSES
ADDRESSES:0:2863
kafka-run-class.sh kafka.tools.GetOffsetShell — broker-list ogg21cbigdata:9092 — topic ORDER_ITEMS
ORDER_ITEMS:0:22738