Package com.zendesk.maxwell
Class MaxwellConfig
- java.lang.Object
-
- com.zendesk.maxwell.util.AbstractConfig
-
- com.zendesk.maxwell.MaxwellConfig
-
public class MaxwellConfig extends AbstractConfig
Configuration object for Maxwell
-
-
Field Summary
Fields Modifier and Type Field Description StringbigQueryDatasetMaxwellBigQueryProducerdatasetStringbigQueryProjectIdMaxwellBigQueryProducerproject idStringbigQueryTableMaxwellBigQueryProducertableintbinlogEventQueueSizeThe size for the queue used to buffer events parsed off binlog inBinlogConnectorReplicatorStringbootstrapperType"async" or "sync", describes bootstrapping behaviorintbufferedProducerSizesize of queue for buffered producerfloatbufferMemoryUsagefloat between 0 and 1, defines percentage of JVM memory to use buffering rows.StringclientIDString that uniquely identifies this instance of maxwellMaxwellHealthCheckFactorycustomHealthFactoryAvailable to customer producers for configuration.PropertiescustomProducerPropertiesAvailable to customer producers for configuration.StringdatabaseNameName of database in which to store maxwell data (default `maxwell`)StringddlKafkaTopicKafka topic to send schema changes (DDL) toStringddlPubsubTopicMaxwellPubsubProducerDDL topicStringdeadLetterTopicKafka topic to send undeliverable rows toMaxwellDiagnosticContext.ConfigdiagnosticConfigconfiguration for maxwell http diagnostic endpointbooleanenableHttpConfigwhether to enable reconfiguration via http endpointStringexcludeColumnsfilter out these columnsFilterfilterConfiguration for including/excluding rowsStringfilterListMaxwell filtersstatic StringGTID_MODE_ENVString that describes an environment key that, if set, will enable Maxwell's GTID modeBooleangtidModeAttempt to use Mysql GTIDs to keep track of positionbooleanhaModeEnable high available support (via jgroups-raft)com.codahale.metrics.health.HealthCheckRegistryhealthCheckRegistrycontainer for maxwell health checksStringhttpBindAddressbind adress for metrics/admin serverStringhttpPathPrefixpath prefix for metrics/admin serverinthttpPorthttp port for metrics/admin serverBooleanignoreMissingSchemaIgnore any missing database / table schemas, unless they're included as part of filters.booleanignoreProducerErrorIf true, continue on certain producer errors.PositioninitPositionOverride Maxwell's stored starting positionStringjavascriptFilepath to file containing javascript filtering functionsStringjgroupsConfPath to raft.xml file that configures high availability supportStringkafkaKeyFormat"hash" or "array" -- defines format of kafka keyStringkafkaPartitionHash"default" or "murmur3", defines partition-choice hash functionPropertieskafkaPropertiesProperties object containing all configuration options beginning with "kafka."StringkafkaTopicMain kafka topic to produce tobooleankinesisMd5KeysIf true, pass key throughDigestUtils.md5Hex(byte[])before sending to Kinesis.
Limits the size of the kinesis key, iirc.StringkinesisStreamKinesis stream nameStringlog_levelstring representation of java log levelbooleanmasterRecoveryEnable non-GTID master recovery codeIntegermaxSchemaDeltasHow many schema "deltas" are kept live before a schema compaction is triggered.MaxwellMysqlConfigmaxwellMysqlSpecify a "root" maxwell servercom.codahale.metrics.MetricRegistrymetricRegistrycontainer for maxwell metric collectionintmetricsAgeSlotime in seconds before incrementing the "slo_violation" metricStringmetricsDatadogAPIKeydatadog apikey used when reporting type is httpStringmetricsDatadogHosthost to send UDP DD metrics toLongmetricsDatadogIntervaltime in seconds between datadog metrics pushesintmetricsDatadogPortport to send UDP DD metrics toStringmetricsDatadogSite"us" or "eu"StringmetricsDatadogTagslist of additional tags to send to datadog, as tag:value,tag:valueStringmetricsDatadogTypeHow to report metrics to datadog, either "udp" or "http"booleanmetricsJvmwhether to report JVM metricsStringmetricsPrefixpath prefix for metrics serverStringmetricsReportingTypestring describing how to report metrics.LongmetricsSlf4jIntervalfor slf4j metrics reporter, how often to reportStringnatsSubjectNatsProducerMessage SubjectStringnatsUrlNatsProducerURLMaxwellOutputConfigoutputConfigControls output features and formatsStringoutputFileoutput file path for theFileProducerLongproducerAckTimeoutUsed in all producers deriving fromAbstractAsyncProducer.
In milliseconds, time a message can spend in theInflightMessageListwithout server acknowledgement before being considered lost.ProducerFactoryproducerFactoryIf non-null, generate a producer with this factoryStringproducerPartitionColumnswhen producerPartitionKey is "column", list of columns to partition byStringproducerPartitionFallbackwhen producerPartitionKey is "column", database|table|primary_key to fall back to
(when column is unavailable)StringproducerPartitionKeydatabase|table|primary_key|transaction_id|column|random
Input for partition choice functionStringproducerTypeString describing desired producer type: "kafka", "kinesis", etc.StringpubsubEmulatorMaxwellPubsubProduceremulator host to use, if specifiedorg.threeten.bp.DurationpubsubInitialRpcTimeoutMaxwellPubsubProducerinitial RPC timeoutorg.threeten.bp.DurationpubsubMaxRetryDelayMaxwellPubsubProducermax retry delayorg.threeten.bp.DurationpubsubMaxRpcTimeoutMaxwellPubsubProducermax RPC timeoutLongpubsubMessageCountBatchSizeMaxwellPubsubProducermessage count batch sizeStringpubsubMessageOrderingKeyMaxwellPubsubProducermessage ordering key template (will enable message ordering if specified)StringpubsubProjectIdMaxwellPubsubProducerproject idorg.threeten.bp.DurationpubsubPublishDelayThresholdMaxwellPubsubProducerpublish delay thresholdLongpubsubRequestBytesThresholdMaxwellPubsubProducerbytes request thresholdorg.threeten.bp.DurationpubsubRetryDelayMaxwellPubsubProducerretry delayFloatpubsubRetryDelayMultiplierMaxwellPubsubProducerretry delay multiplierFloatpubsubRpcTimeoutMultiplierMaxwellPubsubProducerRPC timeout multiplierStringpubsubTopicMaxwellPubsubProducertopicorg.threeten.bp.DurationpubsubTotalTimeoutMaxwellPubsubProducertotal timeoutbooleanrabbitmqDeclareExchangeRabbitmqProducerdeclare exchangeStringrabbitmqExchangeRabbitmqProducerexchangebooleanrabbitMqExchangeAutoDeleteRabbitmqProducerexchange audo deletionbooleanrabbitMqExchangeDurableRabbitmqProducerexchange durabilityStringrabbitmqExchangeTypeRabbitmqProducerexchange typeIntegerrabbitmqHandshakeTimeoutRabbitmqProducerhandshake timeoutStringrabbitmqHostRabbitmqProducerhostbooleanrabbitmqMessagePersistentRabbitmqProducermessage persistenceStringrabbitmqPassRabbitmqProducerpasswordIntegerrabbitmqPortRabbitmqProducerportStringrabbitmqRoutingKeyTemplateRabbitmqProducerrouting key templateStringrabbitmqURIRabbitmqProducerurl (alternative to other configuration settings)StringrabbitmqUserRabbitmqProducerusernameStringrabbitmqVirtualHostRabbitmqProducervirtual hostStringraftMemberIDDefines membership within a HA clusterbooleanrecaptureSchemaForce a new schema capture upon startup.StringredisAuthMaxwellRedisProducerpasswordintredisDatabaseMaxwellRedisProducerdatabaseStringredisHostMaxwellRedisProducerhostStringredisKeyintredisPortMaxwellRedisProducerportStringredisSentinelMasterNameMaxwellRedisProducername of master redis sentinelStringredisSentinelsMaxwellRedisProducercomma seperated list of redis sentialsStringredisStreamJsonKeyMaxwellRedisProducerJSON key for XADDStringredisTypetype of redis operation to perform: XADD, LPUSH, RPUSH, PUBSUBbooleanreplayModeIf true, Maxwell plays events but otherwise stores no schema changes or position changesLongreplicaServerIDinteger that maxwell will report to the server as its "server_id".MaxwellMysqlConfigreplicationMysqlIf non-null, specify a mysql to replicate from.
If non, fallback tomaxwellMysqlintreplicationReconnectionRetriesNumber of times to attempt connecting the replicator before giving upMaxwellMysqlConfigschemaMysqlIf non-null, specify a mysql server to capture schema from If non, fallback tomaxwellMysqlScriptingscriptingInstantiated byvalidate().StringsnsAttrsMaxwellSQSProducer["table"|"database"] -- if set, interpolate either/or table / database into the messageStringsnsTopicMaxwellSQSProducertopicStringsqsQueueUriMaxwellSQSProducerQueue URIStringsqsServiceEndpointMaxwellSQSProducerQueue Service Endpoint URLStringsqsSigningRegionMaxwellSQSProducerQueue Signing region-
Fields inherited from class com.zendesk.maxwell.util.AbstractConfig
DEFAULT_CONFIG_FILE
-
-
Constructor Summary
Constructors Constructor Description MaxwellConfig()Build a default configuration object.MaxwellConfig(String[] argv)build a configuration instance from command line arguments
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected MaxwellOptionParserbuildOptionParser()protected MaxwellHealthCheckFactoryfetchHealthCheckFactory(joptsimple.OptionSet options, Properties properties)If present in the configuration, build an instance of a custom health factoryprotected ProducerFactoryfetchProducerFactory(joptsimple.OptionSet options, Properties properties)If present in the configuration, build an instance of a custom producer factorBooleangetIgnoreMissingSchema()PropertiesgetKafkaProperties()return a filtered list of properties for the Kafka producervoidsetIgnoreMissingSchema(Boolean ignoreMissingSchema)voidvalidate()Validate the maxwell configuration, exiting with an error message if invalid.-
Methods inherited from class com.zendesk.maxwell.util.AbstractConfig
fetchBooleanOption, fetchFloatOption, fetchIntegerOption, fetchLongOption, fetchOption, fetchStringOption, parseMysqlConfig, readPropertiesEnv, readPropertiesFile, usage, usage, usageForOptions
-
-
-
-
Field Detail
-
GTID_MODE_ENV
public static final String GTID_MODE_ENV
String that describes an environment key that, if set, will enable Maxwell's GTID modePrimarily used for test environment setup.
- See Also:
- Constant Field Values
-
replicationMysql
public MaxwellMysqlConfig replicationMysql
If non-null, specify a mysql to replicate from.
If non, fallback tomaxwellMysql
-
replicationReconnectionRetries
public int replicationReconnectionRetries
Number of times to attempt connecting the replicator before giving up
-
schemaMysql
public MaxwellMysqlConfig schemaMysql
If non-null, specify a mysql server to capture schema from If non, fallback tomaxwellMysql
-
maxwellMysql
public MaxwellMysqlConfig maxwellMysql
Specify a "root" maxwell server
-
filter
public Filter filter
Configuration for including/excluding rows
-
ignoreMissingSchema
public Boolean ignoreMissingSchema
Ignore any missing database / table schemas, unless they're included as part of filters. Default false. Don't use unless you really really need to.
-
gtidMode
public Boolean gtidMode
Attempt to use Mysql GTIDs to keep track of position
-
databaseName
public String databaseName
Name of database in which to store maxwell data (default `maxwell`)
-
excludeColumns
public String excludeColumns
filter out these columns
-
filterList
public String filterList
Maxwell filters
-
producerFactory
public ProducerFactory producerFactory
If non-null, generate a producer with this factory
-
customProducerProperties
public final Properties customProducerProperties
Available to customer producers for configuration. Setup with all properties prefixed `customer_producer.`
-
customHealthFactory
public MaxwellHealthCheckFactory customHealthFactory
Available to customer producers for configuration. Setup with all properties prefixed `customer_producer.`
-
producerType
public String producerType
String describing desired producer type: "kafka", "kinesis", etc.
-
kafkaProperties
public final Properties kafkaProperties
Properties object containing all configuration options beginning with "kafka."
-
kafkaTopic
public String kafkaTopic
Main kafka topic to produce to
-
deadLetterTopic
public String deadLetterTopic
Kafka topic to send undeliverable rows to
-
ddlKafkaTopic
public String ddlKafkaTopic
Kafka topic to send schema changes (DDL) to
-
kafkaKeyFormat
public String kafkaKeyFormat
"hash" or "array" -- defines format of kafka key
-
kafkaPartitionHash
public String kafkaPartitionHash
"default" or "murmur3", defines partition-choice hash function
-
bootstrapperType
public String bootstrapperType
"async" or "sync", describes bootstrapping behavior
-
bufferedProducerSize
public int bufferedProducerSize
size of queue for buffered producer
-
producerPartitionKey
public String producerPartitionKey
database|table|primary_key|transaction_id|column|random
Input for partition choice function
-
producerPartitionColumns
public String producerPartitionColumns
when producerPartitionKey is "column", list of columns to partition by
-
producerPartitionFallback
public String producerPartitionFallback
when producerPartitionKey is "column", database|table|primary_key to fall back to
(when column is unavailable)
-
kinesisStream
public String kinesisStream
Kinesis stream name
-
kinesisMd5Keys
public boolean kinesisMd5Keys
If true, pass key throughDigestUtils.md5Hex(byte[])before sending to Kinesis.
Limits the size of the kinesis key, iirc.
-
sqsQueueUri
public String sqsQueueUri
MaxwellSQSProducerQueue URI
-
sqsServiceEndpoint
public String sqsServiceEndpoint
MaxwellSQSProducerQueue Service Endpoint URL
-
sqsSigningRegion
public String sqsSigningRegion
MaxwellSQSProducerQueue Signing region
-
snsTopic
public String snsTopic
MaxwellSQSProducertopic
-
snsAttrs
public String snsAttrs
MaxwellSQSProducer["table"|"database"] -- if set, interpolate either/or table / database into the message
-
pubsubProjectId
public String pubsubProjectId
MaxwellPubsubProducerproject id
-
pubsubTopic
public String pubsubTopic
MaxwellPubsubProducertopic
-
ddlPubsubTopic
public String ddlPubsubTopic
MaxwellPubsubProducerDDL topic
-
pubsubRequestBytesThreshold
public Long pubsubRequestBytesThreshold
MaxwellPubsubProducerbytes request threshold
-
pubsubMessageCountBatchSize
public Long pubsubMessageCountBatchSize
MaxwellPubsubProducermessage count batch size
-
pubsubMessageOrderingKey
public String pubsubMessageOrderingKey
MaxwellPubsubProducermessage ordering key template (will enable message ordering if specified)
-
pubsubPublishDelayThreshold
public org.threeten.bp.Duration pubsubPublishDelayThreshold
MaxwellPubsubProducerpublish delay threshold
-
pubsubRetryDelay
public org.threeten.bp.Duration pubsubRetryDelay
MaxwellPubsubProducerretry delay
-
pubsubRetryDelayMultiplier
public Float pubsubRetryDelayMultiplier
MaxwellPubsubProducerretry delay multiplier
-
pubsubMaxRetryDelay
public org.threeten.bp.Duration pubsubMaxRetryDelay
MaxwellPubsubProducermax retry delay
-
pubsubInitialRpcTimeout
public org.threeten.bp.Duration pubsubInitialRpcTimeout
MaxwellPubsubProducerinitial RPC timeout
-
pubsubRpcTimeoutMultiplier
public Float pubsubRpcTimeoutMultiplier
MaxwellPubsubProducerRPC timeout multiplier
-
pubsubMaxRpcTimeout
public org.threeten.bp.Duration pubsubMaxRpcTimeout
MaxwellPubsubProducermax RPC timeout
-
pubsubTotalTimeout
public org.threeten.bp.Duration pubsubTotalTimeout
MaxwellPubsubProducertotal timeout
-
pubsubEmulator
public String pubsubEmulator
MaxwellPubsubProduceremulator host to use, if specified
-
bigQueryProjectId
public String bigQueryProjectId
MaxwellBigQueryProducerproject id
-
bigQueryDataset
public String bigQueryDataset
MaxwellBigQueryProducerdataset
-
bigQueryTable
public String bigQueryTable
MaxwellBigQueryProducertable
-
producerAckTimeout
public Long producerAckTimeout
Used in all producers deriving fromAbstractAsyncProducer.
In milliseconds, time a message can spend in theInflightMessageListwithout server acknowledgement before being considered lost.
-
outputFile
public String outputFile
output file path for theFileProducer
-
outputConfig
public MaxwellOutputConfig outputConfig
Controls output features and formats
-
log_level
public String log_level
string representation of java log level
-
metricRegistry
public com.codahale.metrics.MetricRegistry metricRegistry
container for maxwell metric collection
-
healthCheckRegistry
public com.codahale.metrics.health.HealthCheckRegistry healthCheckRegistry
container for maxwell health checks
-
httpPort
public int httpPort
http port for metrics/admin server
-
httpBindAddress
public String httpBindAddress
bind adress for metrics/admin server
-
httpPathPrefix
public String httpPathPrefix
path prefix for metrics/admin server
-
metricsPrefix
public String metricsPrefix
path prefix for metrics server
-
metricsReportingType
public String metricsReportingType
string describing how to report metrics.
-
metricsSlf4jInterval
public Long metricsSlf4jInterval
for slf4j metrics reporter, how often to report
-
metricsDatadogType
public String metricsDatadogType
How to report metrics to datadog, either "udp" or "http"
-
metricsDatadogTags
public String metricsDatadogTags
list of additional tags to send to datadog, as tag:value,tag:value
-
metricsDatadogAPIKey
public String metricsDatadogAPIKey
datadog apikey used when reporting type is http
-
metricsDatadogSite
public String metricsDatadogSite
"us" or "eu"
-
metricsDatadogHost
public String metricsDatadogHost
host to send UDP DD metrics to
-
metricsDatadogPort
public int metricsDatadogPort
port to send UDP DD metrics to
-
metricsDatadogInterval
public Long metricsDatadogInterval
time in seconds between datadog metrics pushes
-
metricsJvm
public boolean metricsJvm
whether to report JVM metrics
-
metricsAgeSlo
public int metricsAgeSlo
time in seconds before incrementing the "slo_violation" metric
-
diagnosticConfig
public MaxwellDiagnosticContext.Config diagnosticConfig
configuration for maxwell http diagnostic endpoint
-
enableHttpConfig
public boolean enableHttpConfig
whether to enable reconfiguration via http endpointFor the moment this endpoint only allows changing of filters in runtime
-
clientID
public String clientID
String that uniquely identifies this instance of maxwell
-
replicaServerID
public Long replicaServerID
integer that maxwell will report to the server as its "server_id".Must be unique within the cluster.
-
initPosition
public Position initPosition
Override Maxwell's stored starting position
-
replayMode
public boolean replayMode
If true, Maxwell plays events but otherwise stores no schema changes or position changes
-
masterRecovery
public boolean masterRecovery
Enable non-GTID master recovery code
-
ignoreProducerError
public boolean ignoreProducerError
If true, continue on certain producer errors. Otherwise crash.
-
recaptureSchema
public boolean recaptureSchema
Force a new schema capture upon startup. dangerous.
-
bufferMemoryUsage
public float bufferMemoryUsage
float between 0 and 1, defines percentage of JVM memory to use buffering rows.actual formula is given as bufferMemoryUsage * Runtime.getRuntime().maxMemory().
-
maxSchemaDeltas
public Integer maxSchemaDeltas
How many schema "deltas" are kept live before a schema compaction is triggered.- See Also:
MysqlSchemaCompactor
-
rabbitmqUser
public String rabbitmqUser
RabbitmqProducerusername
-
rabbitmqPass
public String rabbitmqPass
RabbitmqProducerpassword
-
rabbitmqHost
public String rabbitmqHost
RabbitmqProducerhost
-
rabbitmqPort
public Integer rabbitmqPort
RabbitmqProducerport
-
rabbitmqVirtualHost
public String rabbitmqVirtualHost
RabbitmqProducervirtual host
-
rabbitmqURI
public String rabbitmqURI
RabbitmqProducerurl (alternative to other configuration settings)
-
rabbitmqHandshakeTimeout
public Integer rabbitmqHandshakeTimeout
RabbitmqProducerhandshake timeout
-
rabbitmqExchange
public String rabbitmqExchange
RabbitmqProducerexchange
-
rabbitmqExchangeType
public String rabbitmqExchangeType
RabbitmqProducerexchange type
-
rabbitMqExchangeDurable
public boolean rabbitMqExchangeDurable
RabbitmqProducerexchange durability
-
rabbitMqExchangeAutoDelete
public boolean rabbitMqExchangeAutoDelete
RabbitmqProducerexchange audo deletion
-
rabbitmqRoutingKeyTemplate
public String rabbitmqRoutingKeyTemplate
RabbitmqProducerrouting key template
-
rabbitmqMessagePersistent
public boolean rabbitmqMessagePersistent
RabbitmqProducermessage persistence
-
rabbitmqDeclareExchange
public boolean rabbitmqDeclareExchange
RabbitmqProducerdeclare exchange
-
natsUrl
public String natsUrl
NatsProducerURL
-
natsSubject
public String natsSubject
NatsProducerMessage Subject
-
redisHost
public String redisHost
MaxwellRedisProducerhost
-
redisPort
public int redisPort
MaxwellRedisProducerport
-
redisAuth
public String redisAuth
MaxwellRedisProducerpassword
-
redisDatabase
public int redisDatabase
MaxwellRedisProducerdatabase
-
redisKey
public String redisKey
-
redisStreamJsonKey
public String redisStreamJsonKey
MaxwellRedisProducerJSON key for XADDwhen XADD is used, the event is embedded as a JSON string inside a field named this. defaults to 'message'
-
redisSentinels
public String redisSentinels
MaxwellRedisProducercomma seperated list of redis sentials
-
redisSentinelMasterName
public String redisSentinelMasterName
MaxwellRedisProducername of master redis sentinel
-
redisType
public String redisType
type of redis operation to perform: XADD, LPUSH, RPUSH, PUBSUB
-
javascriptFile
public String javascriptFile
path to file containing javascript filtering functions
-
scripting
public Scripting scripting
Instantiated byvalidate(). Should be moved to MaxwellContext.
-
haMode
public boolean haMode
Enable high available support (via jgroups-raft)
-
jgroupsConf
public String jgroupsConf
Path to raft.xml file that configures high availability support
-
raftMemberID
public String raftMemberID
Defines membership within a HA cluster
-
binlogEventQueueSize
public int binlogEventQueueSize
The size for the queue used to buffer events parsed off binlog inBinlogConnectorReplicator
-
-
Constructor Detail
-
MaxwellConfig
public MaxwellConfig()
Build a default configuration object.
-
MaxwellConfig
public MaxwellConfig(String[] argv)
build a configuration instance from command line arguments- Parameters:
argv- command line arguments
-
-
Method Detail
-
buildOptionParser
protected MaxwellOptionParser buildOptionParser()
- Specified by:
buildOptionParserin classAbstractConfig
-
validate
public void validate()
Validate the maxwell configuration, exiting with an error message if invalid.
-
getKafkaProperties
public Properties getKafkaProperties()
return a filtered list of properties for the Kafka producer- Returns:
- Properties object containing all kafka properties found in config.properties
-
fetchProducerFactory
protected ProducerFactory fetchProducerFactory(joptsimple.OptionSet options, Properties properties)
If present in the configuration, build an instance of a custom producer factor- Parameters:
options- command line argumentsproperties- properties from config.properties- Returns:
- NULL or ProducerFactory instance
-
fetchHealthCheckFactory
protected MaxwellHealthCheckFactory fetchHealthCheckFactory(joptsimple.OptionSet options, Properties properties)
If present in the configuration, build an instance of a custom health factory- Parameters:
options- command line argumentsproperties- properties from config.properties- Returns:
- NULL or MaxwellHealthCheckFactory instance
-
getIgnoreMissingSchema
public Boolean getIgnoreMissingSchema()
-
setIgnoreMissingSchema
public void setIgnoreMissingSchema(Boolean ignoreMissingSchema)
-
-