Package com.zendesk.maxwell
Class MaxwellConfig
- java.lang.Object
-
- com.zendesk.maxwell.util.AbstractConfig
-
- com.zendesk.maxwell.MaxwellConfig
-
public class MaxwellConfig extends AbstractConfig
Configuration object for Maxwell
-
-
Field Summary
Fields Modifier and Type Field Description String
bigQueryDataset
MaxwellBigQueryProducer
datasetString
bigQueryProjectId
MaxwellBigQueryProducer
project idString
bigQueryTable
MaxwellBigQueryProducer
tableint
binlogEventQueueSize
The size for the queue used to buffer events parsed off binlog inBinlogConnectorReplicator
String
bootstrapperType
"async" or "sync", describes bootstrapping behaviorint
bufferedProducerSize
size of queue for buffered producerfloat
bufferMemoryUsage
float between 0 and 1, defines percentage of JVM memory to use buffering rows.String
clientID
String that uniquely identifies this instance of maxwellMaxwellHealthCheckFactory
customHealthFactory
Available to customer producers for configuration.Properties
customProducerProperties
Available to customer producers for configuration.String
databaseName
Name of database in which to store maxwell data (default `maxwell`)String
ddlKafkaTopic
Kafka topic to send schema changes (DDL) toString
ddlPubsubTopic
MaxwellPubsubProducer
DDL topicString
deadLetterTopic
Kafka topic to send undeliverable rows toMaxwellDiagnosticContext.Config
diagnosticConfig
configuration for maxwell http diagnostic endpointboolean
enableHttpConfig
whether to enable reconfiguration via http endpointString
excludeColumns
filter out these columnsFilter
filter
Configuration for including/excluding rowsString
filterList
Maxwell filtersstatic String
GTID_MODE_ENV
String that describes an environment key that, if set, will enable Maxwell's GTID modeBoolean
gtidMode
Attempt to use Mysql GTIDs to keep track of positionboolean
haMode
Enable high available support (via jgroups-raft)com.codahale.metrics.health.HealthCheckRegistry
healthCheckRegistry
container for maxwell health checksString
httpBindAddress
bind adress for metrics/admin serverString
httpPathPrefix
path prefix for metrics/admin serverint
httpPort
http port for metrics/admin serverBoolean
ignoreMissingSchema
Ignore any missing database / table schemas, unless they're included as part of filters.boolean
ignoreProducerError
If true, continue on certain producer errors.Position
initPosition
Override Maxwell's stored starting positionString
javascriptFile
path to file containing javascript filtering functionsString
jgroupsConf
Path to raft.xml file that configures high availability supportString
kafkaKeyFormat
"hash" or "array" -- defines format of kafka keyString
kafkaPartitionHash
"default" or "murmur3", defines partition-choice hash functionProperties
kafkaProperties
Properties object containing all configuration options beginning with "kafka."String
kafkaTopic
Main kafka topic to produce toboolean
kinesisMd5Keys
If true, pass key throughDigestUtils.md5Hex(byte[])
before sending to Kinesis.
Limits the size of the kinesis key, iirc.String
kinesisStream
Kinesis stream nameString
log_level
string representation of java log levelboolean
masterRecovery
Enable non-GTID master recovery codeInteger
maxSchemaDeltas
How many schema "deltas" are kept live before a schema compaction is triggered.MaxwellMysqlConfig
maxwellMysql
Specify a "root" maxwell servercom.codahale.metrics.MetricRegistry
metricRegistry
container for maxwell metric collectionint
metricsAgeSlo
time in seconds before incrementing the "slo_violation" metricString
metricsDatadogAPIKey
datadog apikey used when reporting type is httpString
metricsDatadogHost
host to send UDP DD metrics toLong
metricsDatadogInterval
time in seconds between datadog metrics pushesint
metricsDatadogPort
port to send UDP DD metrics toString
metricsDatadogSite
"us" or "eu"String
metricsDatadogTags
list of additional tags to send to datadog, as tag:value,tag:valueString
metricsDatadogType
How to report metrics to datadog, either "udp" or "http"boolean
metricsJvm
whether to report JVM metricsString
metricsPrefix
path prefix for metrics serverString
metricsReportingType
string describing how to report metrics.Long
metricsSlf4jInterval
for slf4j metrics reporter, how often to reportString
natsSubject
NatsProducer
Message SubjectString
natsUrl
NatsProducer
URLMaxwellOutputConfig
outputConfig
Controls output features and formatsString
outputFile
output file path for theFileProducer
Long
producerAckTimeout
Used in all producers deriving fromAbstractAsyncProducer
.
In milliseconds, time a message can spend in theInflightMessageList
without server acknowledgement before being considered lost.ProducerFactory
producerFactory
If non-null, generate a producer with this factoryString
producerPartitionColumns
when producerPartitionKey is "column", list of columns to partition byString
producerPartitionFallback
when producerPartitionKey is "column", database|table|primary_key to fall back to
(when column is unavailable)String
producerPartitionKey
database|table|primary_key|transaction_id|column|random
Input for partition choice functionString
producerType
String describing desired producer type: "kafka", "kinesis", etc.String
pubsubEmulator
MaxwellPubsubProducer
emulator host to use, if specifiedorg.threeten.bp.Duration
pubsubInitialRpcTimeout
MaxwellPubsubProducer
initial RPC timeoutorg.threeten.bp.Duration
pubsubMaxRetryDelay
MaxwellPubsubProducer
max retry delayorg.threeten.bp.Duration
pubsubMaxRpcTimeout
MaxwellPubsubProducer
max RPC timeoutLong
pubsubMessageCountBatchSize
MaxwellPubsubProducer
message count batch sizeString
pubsubMessageOrderingKey
MaxwellPubsubProducer
message ordering key template (will enable message ordering if specified)String
pubsubProjectId
MaxwellPubsubProducer
project idorg.threeten.bp.Duration
pubsubPublishDelayThreshold
MaxwellPubsubProducer
publish delay thresholdLong
pubsubRequestBytesThreshold
MaxwellPubsubProducer
bytes request thresholdorg.threeten.bp.Duration
pubsubRetryDelay
MaxwellPubsubProducer
retry delayFloat
pubsubRetryDelayMultiplier
MaxwellPubsubProducer
retry delay multiplierFloat
pubsubRpcTimeoutMultiplier
MaxwellPubsubProducer
RPC timeout multiplierString
pubsubTopic
MaxwellPubsubProducer
topicorg.threeten.bp.Duration
pubsubTotalTimeout
MaxwellPubsubProducer
total timeoutboolean
rabbitmqDeclareExchange
RabbitmqProducer
declare exchangeString
rabbitmqExchange
RabbitmqProducer
exchangeboolean
rabbitMqExchangeAutoDelete
RabbitmqProducer
exchange audo deletionboolean
rabbitMqExchangeDurable
RabbitmqProducer
exchange durabilityString
rabbitmqExchangeType
RabbitmqProducer
exchange typeInteger
rabbitmqHandshakeTimeout
RabbitmqProducer
handshake timeoutString
rabbitmqHost
RabbitmqProducer
hostboolean
rabbitmqMessagePersistent
RabbitmqProducer
message persistenceString
rabbitmqPass
RabbitmqProducer
passwordInteger
rabbitmqPort
RabbitmqProducer
portString
rabbitmqRoutingKeyTemplate
RabbitmqProducer
routing key templateString
rabbitmqURI
RabbitmqProducer
url (alternative to other configuration settings)String
rabbitmqUser
RabbitmqProducer
usernameString
rabbitmqVirtualHost
RabbitmqProducer
virtual hostString
raftMemberID
Defines membership within a HA clusterboolean
recaptureSchema
Force a new schema capture upon startup.String
redisAuth
MaxwellRedisProducer
passwordint
redisDatabase
MaxwellRedisProducer
databaseString
redisHost
MaxwellRedisProducer
hostString
redisKey
int
redisPort
MaxwellRedisProducer
portString
redisSentinelMasterName
MaxwellRedisProducer
name of master redis sentinelString
redisSentinels
MaxwellRedisProducer
comma seperated list of redis sentialsString
redisStreamJsonKey
MaxwellRedisProducer
JSON key for XADDString
redisType
type of redis operation to perform: XADD, LPUSH, RPUSH, PUBSUBboolean
replayMode
If true, Maxwell plays events but otherwise stores no schema changes or position changesLong
replicaServerID
integer that maxwell will report to the server as its "server_id".MaxwellMysqlConfig
replicationMysql
If non-null, specify a mysql to replicate from.
If non, fallback tomaxwellMysql
int
replicationReconnectionRetries
Number of times to attempt connecting the replicator before giving upMaxwellMysqlConfig
schemaMysql
If non-null, specify a mysql server to capture schema from If non, fallback tomaxwellMysql
Scripting
scripting
Instantiated byvalidate()
.String
snsAttrs
MaxwellSQSProducer
["table"|"database"] -- if set, interpolate either/or table / database into the messageString
snsTopic
MaxwellSQSProducer
topicString
sqsQueueUri
MaxwellSQSProducer
Queue URIString
sqsServiceEndpoint
MaxwellSQSProducer
Queue Service Endpoint URLString
sqsSigningRegion
MaxwellSQSProducer
Queue Signing region-
Fields inherited from class com.zendesk.maxwell.util.AbstractConfig
DEFAULT_CONFIG_FILE
-
-
Constructor Summary
Constructors Constructor Description MaxwellConfig()
Build a default configuration object.MaxwellConfig(String[] argv)
build a configuration instance from command line arguments
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected MaxwellOptionParser
buildOptionParser()
protected MaxwellHealthCheckFactory
fetchHealthCheckFactory(joptsimple.OptionSet options, Properties properties)
If present in the configuration, build an instance of a custom health factoryprotected ProducerFactory
fetchProducerFactory(joptsimple.OptionSet options, Properties properties)
If present in the configuration, build an instance of a custom producer factorBoolean
getIgnoreMissingSchema()
Properties
getKafkaProperties()
return a filtered list of properties for the Kafka producervoid
setIgnoreMissingSchema(Boolean ignoreMissingSchema)
void
validate()
Validate the maxwell configuration, exiting with an error message if invalid.-
Methods inherited from class com.zendesk.maxwell.util.AbstractConfig
fetchBooleanOption, fetchFloatOption, fetchIntegerOption, fetchLongOption, fetchOption, fetchStringOption, parseMysqlConfig, readPropertiesEnv, readPropertiesFile, usage, usage, usageForOptions
-
-
-
-
Field Detail
-
GTID_MODE_ENV
public static final String GTID_MODE_ENV
String that describes an environment key that, if set, will enable Maxwell's GTID modePrimarily used for test environment setup.
- See Also:
- Constant Field Values
-
replicationMysql
public MaxwellMysqlConfig replicationMysql
If non-null, specify a mysql to replicate from.
If non, fallback tomaxwellMysql
-
replicationReconnectionRetries
public int replicationReconnectionRetries
Number of times to attempt connecting the replicator before giving up
-
schemaMysql
public MaxwellMysqlConfig schemaMysql
If non-null, specify a mysql server to capture schema from If non, fallback tomaxwellMysql
-
maxwellMysql
public MaxwellMysqlConfig maxwellMysql
Specify a "root" maxwell server
-
filter
public Filter filter
Configuration for including/excluding rows
-
ignoreMissingSchema
public Boolean ignoreMissingSchema
Ignore any missing database / table schemas, unless they're included as part of filters. Default false. Don't use unless you really really need to.
-
gtidMode
public Boolean gtidMode
Attempt to use Mysql GTIDs to keep track of position
-
databaseName
public String databaseName
Name of database in which to store maxwell data (default `maxwell`)
-
excludeColumns
public String excludeColumns
filter out these columns
-
filterList
public String filterList
Maxwell filters
-
producerFactory
public ProducerFactory producerFactory
If non-null, generate a producer with this factory
-
customProducerProperties
public final Properties customProducerProperties
Available to customer producers for configuration. Setup with all properties prefixed `customer_producer.`
-
customHealthFactory
public MaxwellHealthCheckFactory customHealthFactory
Available to customer producers for configuration. Setup with all properties prefixed `customer_producer.`
-
producerType
public String producerType
String describing desired producer type: "kafka", "kinesis", etc.
-
kafkaProperties
public final Properties kafkaProperties
Properties object containing all configuration options beginning with "kafka."
-
kafkaTopic
public String kafkaTopic
Main kafka topic to produce to
-
deadLetterTopic
public String deadLetterTopic
Kafka topic to send undeliverable rows to
-
ddlKafkaTopic
public String ddlKafkaTopic
Kafka topic to send schema changes (DDL) to
-
kafkaKeyFormat
public String kafkaKeyFormat
"hash" or "array" -- defines format of kafka key
-
kafkaPartitionHash
public String kafkaPartitionHash
"default" or "murmur3", defines partition-choice hash function
-
bootstrapperType
public String bootstrapperType
"async" or "sync", describes bootstrapping behavior
-
bufferedProducerSize
public int bufferedProducerSize
size of queue for buffered producer
-
producerPartitionKey
public String producerPartitionKey
database|table|primary_key|transaction_id|column|random
Input for partition choice function
-
producerPartitionColumns
public String producerPartitionColumns
when producerPartitionKey is "column", list of columns to partition by
-
producerPartitionFallback
public String producerPartitionFallback
when producerPartitionKey is "column", database|table|primary_key to fall back to
(when column is unavailable)
-
kinesisStream
public String kinesisStream
Kinesis stream name
-
kinesisMd5Keys
public boolean kinesisMd5Keys
If true, pass key throughDigestUtils.md5Hex(byte[])
before sending to Kinesis.
Limits the size of the kinesis key, iirc.
-
sqsQueueUri
public String sqsQueueUri
MaxwellSQSProducer
Queue URI
-
sqsServiceEndpoint
public String sqsServiceEndpoint
MaxwellSQSProducer
Queue Service Endpoint URL
-
sqsSigningRegion
public String sqsSigningRegion
MaxwellSQSProducer
Queue Signing region
-
snsTopic
public String snsTopic
MaxwellSQSProducer
topic
-
snsAttrs
public String snsAttrs
MaxwellSQSProducer
["table"|"database"] -- if set, interpolate either/or table / database into the message
-
pubsubProjectId
public String pubsubProjectId
MaxwellPubsubProducer
project id
-
pubsubTopic
public String pubsubTopic
MaxwellPubsubProducer
topic
-
ddlPubsubTopic
public String ddlPubsubTopic
MaxwellPubsubProducer
DDL topic
-
pubsubRequestBytesThreshold
public Long pubsubRequestBytesThreshold
MaxwellPubsubProducer
bytes request threshold
-
pubsubMessageCountBatchSize
public Long pubsubMessageCountBatchSize
MaxwellPubsubProducer
message count batch size
-
pubsubMessageOrderingKey
public String pubsubMessageOrderingKey
MaxwellPubsubProducer
message ordering key template (will enable message ordering if specified)
-
pubsubPublishDelayThreshold
public org.threeten.bp.Duration pubsubPublishDelayThreshold
MaxwellPubsubProducer
publish delay threshold
-
pubsubRetryDelay
public org.threeten.bp.Duration pubsubRetryDelay
MaxwellPubsubProducer
retry delay
-
pubsubRetryDelayMultiplier
public Float pubsubRetryDelayMultiplier
MaxwellPubsubProducer
retry delay multiplier
-
pubsubMaxRetryDelay
public org.threeten.bp.Duration pubsubMaxRetryDelay
MaxwellPubsubProducer
max retry delay
-
pubsubInitialRpcTimeout
public org.threeten.bp.Duration pubsubInitialRpcTimeout
MaxwellPubsubProducer
initial RPC timeout
-
pubsubRpcTimeoutMultiplier
public Float pubsubRpcTimeoutMultiplier
MaxwellPubsubProducer
RPC timeout multiplier
-
pubsubMaxRpcTimeout
public org.threeten.bp.Duration pubsubMaxRpcTimeout
MaxwellPubsubProducer
max RPC timeout
-
pubsubTotalTimeout
public org.threeten.bp.Duration pubsubTotalTimeout
MaxwellPubsubProducer
total timeout
-
pubsubEmulator
public String pubsubEmulator
MaxwellPubsubProducer
emulator host to use, if specified
-
bigQueryProjectId
public String bigQueryProjectId
MaxwellBigQueryProducer
project id
-
bigQueryDataset
public String bigQueryDataset
MaxwellBigQueryProducer
dataset
-
bigQueryTable
public String bigQueryTable
MaxwellBigQueryProducer
table
-
producerAckTimeout
public Long producerAckTimeout
Used in all producers deriving fromAbstractAsyncProducer
.
In milliseconds, time a message can spend in theInflightMessageList
without server acknowledgement before being considered lost.
-
outputFile
public String outputFile
output file path for theFileProducer
-
outputConfig
public MaxwellOutputConfig outputConfig
Controls output features and formats
-
log_level
public String log_level
string representation of java log level
-
metricRegistry
public com.codahale.metrics.MetricRegistry metricRegistry
container for maxwell metric collection
-
healthCheckRegistry
public com.codahale.metrics.health.HealthCheckRegistry healthCheckRegistry
container for maxwell health checks
-
httpPort
public int httpPort
http port for metrics/admin server
-
httpBindAddress
public String httpBindAddress
bind adress for metrics/admin server
-
httpPathPrefix
public String httpPathPrefix
path prefix for metrics/admin server
-
metricsPrefix
public String metricsPrefix
path prefix for metrics server
-
metricsReportingType
public String metricsReportingType
string describing how to report metrics.
-
metricsSlf4jInterval
public Long metricsSlf4jInterval
for slf4j metrics reporter, how often to report
-
metricsDatadogType
public String metricsDatadogType
How to report metrics to datadog, either "udp" or "http"
-
metricsDatadogTags
public String metricsDatadogTags
list of additional tags to send to datadog, as tag:value,tag:value
-
metricsDatadogAPIKey
public String metricsDatadogAPIKey
datadog apikey used when reporting type is http
-
metricsDatadogSite
public String metricsDatadogSite
"us" or "eu"
-
metricsDatadogHost
public String metricsDatadogHost
host to send UDP DD metrics to
-
metricsDatadogPort
public int metricsDatadogPort
port to send UDP DD metrics to
-
metricsDatadogInterval
public Long metricsDatadogInterval
time in seconds between datadog metrics pushes
-
metricsJvm
public boolean metricsJvm
whether to report JVM metrics
-
metricsAgeSlo
public int metricsAgeSlo
time in seconds before incrementing the "slo_violation" metric
-
diagnosticConfig
public MaxwellDiagnosticContext.Config diagnosticConfig
configuration for maxwell http diagnostic endpoint
-
enableHttpConfig
public boolean enableHttpConfig
whether to enable reconfiguration via http endpointFor the moment this endpoint only allows changing of filters in runtime
-
clientID
public String clientID
String that uniquely identifies this instance of maxwell
-
replicaServerID
public Long replicaServerID
integer that maxwell will report to the server as its "server_id".Must be unique within the cluster.
-
initPosition
public Position initPosition
Override Maxwell's stored starting position
-
replayMode
public boolean replayMode
If true, Maxwell plays events but otherwise stores no schema changes or position changes
-
masterRecovery
public boolean masterRecovery
Enable non-GTID master recovery code
-
ignoreProducerError
public boolean ignoreProducerError
If true, continue on certain producer errors. Otherwise crash.
-
recaptureSchema
public boolean recaptureSchema
Force a new schema capture upon startup. dangerous.
-
bufferMemoryUsage
public float bufferMemoryUsage
float between 0 and 1, defines percentage of JVM memory to use buffering rows.actual formula is given as bufferMemoryUsage * Runtime.getRuntime().maxMemory().
-
maxSchemaDeltas
public Integer maxSchemaDeltas
How many schema "deltas" are kept live before a schema compaction is triggered.- See Also:
MysqlSchemaCompactor
-
rabbitmqUser
public String rabbitmqUser
RabbitmqProducer
username
-
rabbitmqPass
public String rabbitmqPass
RabbitmqProducer
password
-
rabbitmqHost
public String rabbitmqHost
RabbitmqProducer
host
-
rabbitmqPort
public Integer rabbitmqPort
RabbitmqProducer
port
-
rabbitmqVirtualHost
public String rabbitmqVirtualHost
RabbitmqProducer
virtual host
-
rabbitmqURI
public String rabbitmqURI
RabbitmqProducer
url (alternative to other configuration settings)
-
rabbitmqHandshakeTimeout
public Integer rabbitmqHandshakeTimeout
RabbitmqProducer
handshake timeout
-
rabbitmqExchange
public String rabbitmqExchange
RabbitmqProducer
exchange
-
rabbitmqExchangeType
public String rabbitmqExchangeType
RabbitmqProducer
exchange type
-
rabbitMqExchangeDurable
public boolean rabbitMqExchangeDurable
RabbitmqProducer
exchange durability
-
rabbitMqExchangeAutoDelete
public boolean rabbitMqExchangeAutoDelete
RabbitmqProducer
exchange audo deletion
-
rabbitmqRoutingKeyTemplate
public String rabbitmqRoutingKeyTemplate
RabbitmqProducer
routing key template
-
rabbitmqMessagePersistent
public boolean rabbitmqMessagePersistent
RabbitmqProducer
message persistence
-
rabbitmqDeclareExchange
public boolean rabbitmqDeclareExchange
RabbitmqProducer
declare exchange
-
natsUrl
public String natsUrl
NatsProducer
URL
-
natsSubject
public String natsSubject
NatsProducer
Message Subject
-
redisHost
public String redisHost
MaxwellRedisProducer
host
-
redisPort
public int redisPort
MaxwellRedisProducer
port
-
redisAuth
public String redisAuth
MaxwellRedisProducer
password
-
redisDatabase
public int redisDatabase
MaxwellRedisProducer
database
-
redisKey
public String redisKey
-
redisStreamJsonKey
public String redisStreamJsonKey
MaxwellRedisProducer
JSON key for XADDwhen XADD is used, the event is embedded as a JSON string inside a field named this. defaults to 'message'
-
redisSentinels
public String redisSentinels
MaxwellRedisProducer
comma seperated list of redis sentials
-
redisSentinelMasterName
public String redisSentinelMasterName
MaxwellRedisProducer
name of master redis sentinel
-
redisType
public String redisType
type of redis operation to perform: XADD, LPUSH, RPUSH, PUBSUB
-
javascriptFile
public String javascriptFile
path to file containing javascript filtering functions
-
scripting
public Scripting scripting
Instantiated byvalidate()
. Should be moved to MaxwellContext.
-
haMode
public boolean haMode
Enable high available support (via jgroups-raft)
-
jgroupsConf
public String jgroupsConf
Path to raft.xml file that configures high availability support
-
raftMemberID
public String raftMemberID
Defines membership within a HA cluster
-
binlogEventQueueSize
public int binlogEventQueueSize
The size for the queue used to buffer events parsed off binlog inBinlogConnectorReplicator
-
-
Constructor Detail
-
MaxwellConfig
public MaxwellConfig()
Build a default configuration object.
-
MaxwellConfig
public MaxwellConfig(String[] argv)
build a configuration instance from command line arguments- Parameters:
argv
- command line arguments
-
-
Method Detail
-
buildOptionParser
protected MaxwellOptionParser buildOptionParser()
- Specified by:
buildOptionParser
in classAbstractConfig
-
validate
public void validate()
Validate the maxwell configuration, exiting with an error message if invalid.
-
getKafkaProperties
public Properties getKafkaProperties()
return a filtered list of properties for the Kafka producer- Returns:
- Properties object containing all kafka properties found in config.properties
-
fetchProducerFactory
protected ProducerFactory fetchProducerFactory(joptsimple.OptionSet options, Properties properties)
If present in the configuration, build an instance of a custom producer factor- Parameters:
options
- command line argumentsproperties
- properties from config.properties- Returns:
- NULL or ProducerFactory instance
-
fetchHealthCheckFactory
protected MaxwellHealthCheckFactory fetchHealthCheckFactory(joptsimple.OptionSet options, Properties properties)
If present in the configuration, build an instance of a custom health factory- Parameters:
options
- command line argumentsproperties
- properties from config.properties- Returns:
- NULL or MaxwellHealthCheckFactory instance
-
getIgnoreMissingSchema
public Boolean getIgnoreMissingSchema()
-
setIgnoreMissingSchema
public void setIgnoreMissingSchema(Boolean ignoreMissingSchema)
-
-