Class MaxwellConfig


  • public class MaxwellConfig
    extends AbstractConfig
    Configuration object for Maxwell
    • Field Detail

      • GTID_MODE_ENV

        public static final String GTID_MODE_ENV
        String that describes an environment key that, if set, will enable Maxwell's GTID mode

        Primarily used for test environment setup.

        See Also:
        Constant Field Values
      • replicationReconnectionRetries

        public int replicationReconnectionRetries
        Number of times to attempt connecting the replicator before giving up
      • filter

        public Filter filter
        Configuration for including/excluding rows
      • ignoreMissingSchema

        public Boolean ignoreMissingSchema
        Ignore any missing database / table schemas, unless they're included as part of filters. Default false. Don't use unless you really really need to.
      • gtidMode

        public Boolean gtidMode
        Attempt to use Mysql GTIDs to keep track of position
      • databaseName

        public String databaseName
        Name of database in which to store maxwell data (default `maxwell`)
      • excludeColumns

        public String excludeColumns
        filter out these columns
      • filterList

        public String filterList
        Maxwell filters
      • producerFactory

        public ProducerFactory producerFactory
        If non-null, generate a producer with this factory
      • customProducerProperties

        public final Properties customProducerProperties
        Available to customer producers for configuration. Setup with all properties prefixed `customer_producer.`
      • customHealthFactory

        public MaxwellHealthCheckFactory customHealthFactory
        Available to customer producers for configuration. Setup with all properties prefixed `customer_producer.`
      • producerType

        public String producerType
        String describing desired producer type: "kafka", "kinesis", etc.
      • kafkaProperties

        public final Properties kafkaProperties
        Properties object containing all configuration options beginning with "kafka."
      • kafkaTopic

        public String kafkaTopic
        Main kafka topic to produce to
      • deadLetterTopic

        public String deadLetterTopic
        Kafka topic to send undeliverable rows to
      • ddlKafkaTopic

        public String ddlKafkaTopic
        Kafka topic to send schema changes (DDL) to
      • kafkaKeyFormat

        public String kafkaKeyFormat
        "hash" or "array" -- defines format of kafka key
      • kafkaPartitionHash

        public String kafkaPartitionHash
        "default" or "murmur3", defines partition-choice hash function
      • bootstrapperType

        public String bootstrapperType
        "async" or "sync", describes bootstrapping behavior
      • bufferedProducerSize

        public int bufferedProducerSize
        size of queue for buffered producer
      • producerPartitionKey

        public String producerPartitionKey
        database|table|primary_key|transaction_id|column|random
        Input for partition choice function
      • producerPartitionColumns

        public String producerPartitionColumns
        when producerPartitionKey is "column", list of columns to partition by
      • producerPartitionFallback

        public String producerPartitionFallback
        when producerPartitionKey is "column", database|table|primary_key to fall back to
        (when column is unavailable)
      • kinesisStream

        public String kinesisStream
        Kinesis stream name
      • kinesisMd5Keys

        public boolean kinesisMd5Keys
        If true, pass key through DigestUtils.md5Hex(byte[]) before sending to Kinesis.
        Limits the size of the kinesis key, iirc.
      • snsAttrs

        public String snsAttrs
        MaxwellSQSProducer ["table"|"database"] -- if set, interpolate either/or table / database into the message
      • pubsubMessageOrderingKey

        public String pubsubMessageOrderingKey
        MaxwellPubsubProducer message ordering key template (will enable message ordering if specified)
      • pubsubPublishDelayThreshold

        public org.threeten.bp.Duration pubsubPublishDelayThreshold
        MaxwellPubsubProducer publish delay threshold
      • pubsubMaxRetryDelay

        public org.threeten.bp.Duration pubsubMaxRetryDelay
        MaxwellPubsubProducer max retry delay
      • pubsubInitialRpcTimeout

        public org.threeten.bp.Duration pubsubInitialRpcTimeout
        MaxwellPubsubProducer initial RPC timeout
      • pubsubMaxRpcTimeout

        public org.threeten.bp.Duration pubsubMaxRpcTimeout
        MaxwellPubsubProducer max RPC timeout
      • pubsubTotalTimeout

        public org.threeten.bp.Duration pubsubTotalTimeout
        MaxwellPubsubProducer total timeout
      • producerAckTimeout

        public Long producerAckTimeout
        Used in all producers deriving from AbstractAsyncProducer.
        In milliseconds, time a message can spend in the InflightMessageList without server acknowledgement before being considered lost.
      • log_level

        public String log_level
        string representation of java log level
      • metricRegistry

        public com.codahale.metrics.MetricRegistry metricRegistry
        container for maxwell metric collection
      • healthCheckRegistry

        public com.codahale.metrics.health.HealthCheckRegistry healthCheckRegistry
        container for maxwell health checks
      • httpPort

        public int httpPort
        http port for metrics/admin server
      • httpBindAddress

        public String httpBindAddress
        bind adress for metrics/admin server
      • httpPathPrefix

        public String httpPathPrefix
        path prefix for metrics/admin server
      • metricsPrefix

        public String metricsPrefix
        path prefix for metrics server
      • metricsReportingType

        public String metricsReportingType
        string describing how to report metrics.
      • metricsSlf4jInterval

        public Long metricsSlf4jInterval
        for slf4j metrics reporter, how often to report
      • metricsDatadogType

        public String metricsDatadogType
        How to report metrics to datadog, either "udp" or "http"
      • metricsDatadogTags

        public String metricsDatadogTags
        list of additional tags to send to datadog, as tag:value,tag:value
      • metricsDatadogAPIKey

        public String metricsDatadogAPIKey
        datadog apikey used when reporting type is http
      • metricsDatadogSite

        public String metricsDatadogSite
        "us" or "eu"
      • metricsDatadogHost

        public String metricsDatadogHost
        host to send UDP DD metrics to
      • metricsDatadogPort

        public int metricsDatadogPort
        port to send UDP DD metrics to
      • metricsDatadogInterval

        public Long metricsDatadogInterval
        time in seconds between datadog metrics pushes
      • metricsJvm

        public boolean metricsJvm
        whether to report JVM metrics
      • metricsAgeSlo

        public int metricsAgeSlo
        time in seconds before incrementing the "slo_violation" metric
      • enableHttpConfig

        public boolean enableHttpConfig
        whether to enable reconfiguration via http endpoint

        For the moment this endpoint only allows changing of filters in runtime

      • clientID

        public String clientID
        String that uniquely identifies this instance of maxwell
      • replicaServerID

        public Long replicaServerID
        integer that maxwell will report to the server as its "server_id".

        Must be unique within the cluster.

      • initPosition

        public Position initPosition
        Override Maxwell's stored starting position
      • replayMode

        public boolean replayMode
        If true, Maxwell plays events but otherwise stores no schema changes or position changes
      • masterRecovery

        public boolean masterRecovery
        Enable non-GTID master recovery code
      • ignoreProducerError

        public boolean ignoreProducerError
        If true, continue on certain producer errors. Otherwise crash.
      • recaptureSchema

        public boolean recaptureSchema
        Force a new schema capture upon startup. dangerous.
      • bufferMemoryUsage

        public float bufferMemoryUsage
        float between 0 and 1, defines percentage of JVM memory to use buffering rows.

        actual formula is given as bufferMemoryUsage * Runtime.getRuntime().maxMemory().

      • maxSchemaDeltas

        public Integer maxSchemaDeltas
        How many schema "deltas" are kept live before a schema compaction is triggered.
        See Also:
        MysqlSchemaCompactor
      • rabbitMqExchangeDurable

        public boolean rabbitMqExchangeDurable
        RabbitmqProducer exchange durability
      • rabbitMqExchangeAutoDelete

        public boolean rabbitMqExchangeAutoDelete
        RabbitmqProducer exchange audo deletion
      • rabbitmqMessagePersistent

        public boolean rabbitmqMessagePersistent
        RabbitmqProducer message persistence
      • rabbitmqDeclareExchange

        public boolean rabbitmqDeclareExchange
        RabbitmqProducer declare exchange
      • redisStreamJsonKey

        public String redisStreamJsonKey
        MaxwellRedisProducer JSON key for XADD

        when XADD is used, the event is embedded as a JSON string inside a field named this. defaults to 'message'

      • redisType

        public String redisType
        type of redis operation to perform: XADD, LPUSH, RPUSH, PUBSUB
      • javascriptFile

        public String javascriptFile
        path to file containing javascript filtering functions
      • scripting

        public Scripting scripting
        Instantiated by validate(). Should be moved to MaxwellContext.
      • haMode

        public boolean haMode
        Enable high available support (via jgroups-raft)
      • jgroupsConf

        public String jgroupsConf
        Path to raft.xml file that configures high availability support
      • raftMemberID

        public String raftMemberID
        Defines membership within a HA cluster
      • binlogEventQueueSize

        public int binlogEventQueueSize
        The size for the queue used to buffer events parsed off binlog in BinlogConnectorReplicator
    • Constructor Detail

      • MaxwellConfig

        public MaxwellConfig()
        Build a default configuration object.
      • MaxwellConfig

        public MaxwellConfig​(String[] argv)
        build a configuration instance from command line arguments
        Parameters:
        argv - command line arguments
    • Method Detail

      • validate

        public void validate()
        Validate the maxwell configuration, exiting with an error message if invalid.
      • getKafkaProperties

        public Properties getKafkaProperties()
        return a filtered list of properties for the Kafka producer
        Returns:
        Properties object containing all kafka properties found in config.properties
      • fetchProducerFactory

        protected ProducerFactory fetchProducerFactory​(joptsimple.OptionSet options,
                                                       Properties properties)
        If present in the configuration, build an instance of a custom producer factor
        Parameters:
        options - command line arguments
        properties - properties from config.properties
        Returns:
        NULL or ProducerFactory instance
      • fetchHealthCheckFactory

        protected MaxwellHealthCheckFactory fetchHealthCheckFactory​(joptsimple.OptionSet options,
                                                                    Properties properties)
        If present in the configuration, build an instance of a custom health factory
        Parameters:
        options - command line arguments
        properties - properties from config.properties
        Returns:
        NULL or MaxwellHealthCheckFactory instance
      • getIgnoreMissingSchema

        public Boolean getIgnoreMissingSchema()
      • setIgnoreMissingSchema

        public void setIgnoreMissingSchema​(Boolean ignoreMissingSchema)