CLX

Overview

Xpand provides a utility called CLX to administer distributed SQL.

The primary uses for the utility are:

  • Display status of the Xpand Nodes, disks, or Replica Server.

  • Execute bash or shell commands on Xpand Node(s) in the deployment.

  • Copy data to/from a local file to Xpand Node(s) in the deployment.

  • Issue commands to each Instance's overseeing nanny process.

  • Shutdown the deployment.

  • Start or stop the database.

  • Communicate with a database that’s not online in emergency situations.

Note

If you receive the message, Using cached node details, CLX is relying on cached information that may not reflect the current state of your cluster. Please contact MariaDB Support if it is unexpected or raises concerns.

$ /opt/clustrix/bin/clx status

Options

Option

Description

--db-user (-U)

Login user for SQL connections to the local Xpand Node.

--exclude (-X)

Exclude a specific Xpand Node from this command. May be specified more than once to exclude multiple nodes. Accepts nid or hostname. Takes priority over the --node option.

--help (-h)

Show help message and exit

--node (-n)

Run on a specific Xpand Node, not the entire cluster. May be specified more than once to select multiple nodes. Accepts nid or hostname.

--use-cache (-C)

Use cached cluster information, do not attempt to query the database for this information.

Commands

alters

Provides the status of ALTER statements in progress based on system.alter_progress.

cache

Immediately updates the filesystem cache with node information. Providing this command is typically not necessary as the filesystem cache is saved automatically each time clx connects to a database with all nodes in quorum.

$ clx [options] cache

Command-Specific Option Exceptions:

-n NODE, --node=NODE

Option to run on a specific node has no effect for this command.

-X EXCLUDE, --exclude=EXCLUDE

Option to exclude a specific node has no effect for this command.

cmd

Executes shell command(s) on nodes. Multiple commands may be supplied if enclosed in quotation marks and separated by a semicolon (;) or a pipe (|) character.

$ clx [options] cmd <shell cmd>

Example:

$ clx cmd date
$ clx cmd 'df -h | grep clustrix'

Command-Specific Options:

Option

Description

-i SSH_KEY, --ssh-key=SSH_KEY

SSH Key file to use for private-key auth to other Xpand Nodes. Uses environment variable $CLX_SSH_KEY, if specified. [default: ~/.ssh/id_rsa]

-l SSH_USER, --ssh-user=SSH_USER

Login account for SSH connections to other Xpand Nodes.

Note: non-root users cannot currently use SSH Hostbased Authentication. [default: current user]

-P, --parallel

Execute on all Xpand Nodes simultaneously.

-p SSH_PORT, --ssh-port=SSH_PORT

Port to use for SSH connections to other Xpand Nodes. Uses environment variable $CLX_SSH_PORT, if specified. [default: 22]

-S, --skip-down-nodes

Skip Xpand Nodes that are offline or unavailable instead of aborting when one or more Nodes are unresponsive.

-w, --no-passwd

Do not attempt password SSH authentication.

cp

Executes a control port command on Xpand Nodes.

$ clx [options] cp <cp cmd>

Command-Specific Options:

Option

Description

-P, --parallel

Execute on all Xpand Nodes simultaneously.

-S, --skip-down-nodes

Skip Xpand Nodes that are offline or unavailable instead of aborting when one or more Nodes does not respond.

-T TIMEOUT, --timeout=TIMEOUT

Specify the timeout before failing a Telnet connection (in seconds).

dbrestart

Stops then starts the database process (clxnode) on Xpand Nodes.

$ clx [options] dbrestart

Command-Specific Options:

Option

Description

-T TIMEOUT, --timeout=TIMEOUT

Specify the timeout before failing a Telnet connection (in seconds).

dbstart

Starts the database process (clxnode) on Xpand Nodes.

$ clx [options] dbstart

Command-Specific Options:

Option

Description

-T TIMEOUT, --timeout=TIMEOUT

Specify the timeout before failing a Telnet connection (in seconds).

dbstop

Stops the database process (clxnode) on Xpand Nodes. This effectively shuts down the cluster.

$ clx [options] dbstop

Command-Specific Options:

Option

Description

-T TIMEOUT, --timeout=TIMEOUT

Specify the timeout before failing a Telnet connection (in seconds).

debugdump

Gathers detailed debug information from the cluster to aid troubleshooting by MariaDB Support and Engineering. Xpand assigns an output file if none is specified and displays the name assigned at the end of the process.

$ clx [options] debugdump [outfile]

Command-Specific Options:

Option

Description

--cp-threads=CPTHREADS

Specify how many cp commands to dump in parallel [default: 5 threads]

--overwrite=OVERWRITE

Overwrite an existing file if one exists at the output path.

-S, --skip-down-nodes

Skip nodes that are offline or unavailable instead of aborting when one or more nodes does not respond.

-T TIMEOUT, --timeout=TIMEOUT

Specify the timeout before failing a Telnet connection (in seconds).

-v, --verbose

Display more details.

disks

Displays disk information for the cluster.

$ clx [options] disks

help

Displays help output for clx functions.

$ clx [options] help [command]

logdump

Dumps log files from the cluster. Specify ‘all’ to show clustrix.log, query.log, nanny.log, and debug.log, or list the Xpand and/or system logs that you’d like to see.

logdump also allows Xpand messages file to be selected. If no log file is specified, logdump defaults to clustrix.log. The .log suffix is optional.

$ clx [options] logdump [log files]

Example(s):

$ clx logdump all
$ clx logdump clustrix messages
$ clx logdump query
$ clx --start='2016-09-13' --end='2016-09-14' logdump
$ clx --start='2016-09-13 06' --end= 0 logdump
$ clx --start='2016-09-14' --end=-8 logdump
$ clx --no-sort --skip-down-nodes --limit=2000 logdump

Command-Specific Options:

Option

Description

-a, --archive

Collect individual logfiles into a tar file. This option disables the --limit value and captures everything between --start and --end.

-e END, --end=END

Specify log end time offset in hours after --start (decimals accepted,) or a timestamp string like '2016-02-14 22:51:48' or '2016-02-14 22' (units left off will be zeroed.)

Use a value of 0 (default) to use no end time boundary and read until the end of file. [default: 0 hours]

-i SSH_KEY, --ssh-key=SSH_KEY

SSH Key file to use for private-key auth to other Xpand Nodes. Uses environment variable $CLX_SSH_KEY, if specified. [default: ~/.ssh/id_rsa]

-l SSH_USER, --ssh-user=SSH_USER

Login account for SSH connections to other Xpand Nodes.

Note: non-root users cannot currently use SSH Hostbased Authentication. [default: current user]

-L LIMIT, --limit=LIMIT

Specify the maximum amount (in kB) to collect from each log file. If any file hits this size limit before the specified --start time, the first line from that log will be used in place of the start time, so all logs will be collected from the same time range.

Specify a value of 0 to disable this limit. [default: 10 MiB]. Disabling this limit or setting it too high can starve other processes.

-N, --no-sort

Disable sorting log files together by time, print just one at a time.

-p SSH_PORT, --ssh-port=SSH_PORT

Port to use for SSH connections to other Xpand Nodes. Uses environment variable $CLX_SSH_PORT, if specified. [default: 22]

-s START, --start=START

Specify log start time offset in hours (decimals accepted,) or a timestamp string like '2016-02-14 22:51:48' or '2016-02-14 22' (units left off will be zeroed.)

Use a value of 0 to use no start time boundary and just use the -L size limit. [default: 3 hours]

-S, --skip-down-nodes

Skip Xpand Nodes that are offline or unavailable instead of aborting when one or more Nodes are unresponsive.

-w, --no-passwd

Do not attempt password SSH authentication.

logfreq

Prints a count of matching log lines from the cluster.

$ clx [options] logfreq search_string [log files]

If no log files are specific, this will default to clustrix.log.

Command-specific options:

Option

Description

--case-sensitive

Include this option for case sensitive matching.

-e END, --end=END

Specify log end time offset in hours after --start (decimals accepted,) or a timestamp string like '2016-02-14 22:51:48' or '2016-02-14 22' (units left off will be zeroed.)

Use a value of 0 (default) to use no end time boundary and read until the end of file. [default: 0 hours]

-i SSH_KEY, --ssh-key=SSH_KEY

SSH Key file to use for private-key auth to other Xpand Nodes. Uses environment variable $CLX_SSH_KEY, if specified. [default: ~/.ssh/id_rsa]

-l SSH_USER, --ssh-user=SSH_USER

Login account for SSH connections to other Xpand Nodes.

Note: non-root users cannot currently use SSH Hostbased Authentication. [default: current user]

-N, --no-sort

Disable sorting log files together by time, print just one at a time.

-p SSH_PORT, --ssh-port=SSH_PORT

Port to use for SSH connections to other Xpand Nodes. Uses environment variable $CLX_SSH_PORT, if specified. [default: 22]

-s START, --start=START

Specify log start time offset in hours (decimals accepted,) or a timestamp string like '2016-02-14 22:51:48' or '2016-02-14 22' (units left off will be zeroed.)

Use a value of 0 to use no start time boundary and just use the -L size limit. [default: 3 hours]

-S, --skip-down-nodes

Skip Xpand Nodesthat are offline or unavailable instead of aborting when one or more Nodes does not respond.

-w, --no-passwd

Do not attempt password SSH authentication.

logsearch

Search log files from the cluster.

$ clx [options] logsearch search_string [log files]

If no log files are specific, this will default to clustrix.log.

Command-specific Options:

Option

Description

--case-sensitive

Include this option for case sensitive matching.

-e END, --end=END

Specify log end time offset in hours after --start (decimals accepted,) or a timestamp string like '2016-02-14 22:51:48' or '2016-02-14 22' (units left off will be zeroed.)

Use a value of 0 (default) to use no end time boundary and read until the end of file. [default: 0 hours]

-i SSH_KEY, --ssh-key=SSH_KEY

SSH Key file to use for private-key auth to other Xpand Nodes. Uses environment variable $CLX_SSH_KEY, if specified. [default: ~/.ssh/id_rsa]

-l SSH_USER, --ssh-user=SSH_USER

Login account for SSH connections to other Nodes.

Note: non-root users cannot currently use SSH Hostbased Authentication. [default: current user]

-N, --no-sort

Disable sorting log files together by time, print just one at a time.

-p SSH_PORT, --ssh-port=SSH_PORT

Port to use for SSH connections to other Xpand Nodes. Uses environment variable $CLX_SSH_PORT, if specified. [default: 22]

-s START, --start=START

Specify log start time offset in hours (decimals accepted,) or a timestamp string like '2016-02-14 22:51:48' or '2016-02-14 22' (units left off will be zeroed.)

Use a value of 0 to use no start time boundary and just use the -L size limit. [default: 3 hours]

-S, --skip-down-nodes

Skip Xpand Nodes that are offline or unavailable instead of aborting when one or more Nodes are unresponsive.

-w, --no-passwd

Do not attempt password SSH authentication.

nanny

Executes a nanny command on Xpand Nodes. For the description of the nanny process and the individual nanny commands available, please see the Overview of Nanny.

nanny ensures that all jobs needed for the successful function of Xpand are running. The nanny process is launched automatically by the system init process.

$ clx [options] nanny <nanny cmd>

Example:

$ clx nanny show_jobs

Command-Specific Options:

Option

Description

-P, --parallel

Execute on all nodes simultaneously.

-S, --skip-down-nodes

Skip Xpand Nodes that are offline or unavailable instead of aborting when one or more Nodes does not respond.

-T TIMEOUT, --timeout=TIMEOUT

Specify the timeout before failing a Telnet connection (in seconds).

pubkeyinstall

Uses password auth to generate public keypairs on each Xpand Node and add them to each Node's authorized_keys file.

pull

Copies files from other Xpand Nodes to the local host. This command is only available to root users. If no destination is specified, the current directory name will be used. The nodeid or hostname is appended to the retrieved name.

$ clx [options] pull <remote source> [dest]

Examples:

$ clx pull bar
$ clx pull /var/log/messages ./cluster-logs

Command-specific Options:

Option

Description

-i SSH_KEY, --ssh-key=SSH_KEY

SSH Key file to use for private-key auth to other Xpand Nodes. Uses environment variable $CLX_SSH_KEY, if specified. [default: ~/.ssh/id_rsa]

-l SSH_USER, --ssh-user=SSH_USER

Login account for SSH connections to other nodes.

Note: non-root users cannot currently use SSH Hostbased Authentication. [default: current user]

-p SSH_PORT, --ssh-port=SSH_PORT

Port to use for SSH connections to other Nodes. [default: 22]

-S, --skip-down-nodes

Skip Xpand Nodes that are offline or unavailable instead of aborting when one or more Nodes are unresponsive.

-w, --no-passwd

Do not attempt password SSH authentication.

push

Copies a file from local to Xpand Node. If no destination is specified, the current directory name will be used on all Nodes.

$ clx [options] push <local source> [dest]

Examples:

$ clx push bar
$ clx push /etc/motd /etc

Command-Specific Options:

Option

Description

-i SSH_KEY, --ssh-key=SSH_KEY

SSH Key file to use for private-key auth to other Xpand Nodes. Uses environment variable $CLX_SSH_KEY, if specified. [default: ~/.ssh/id_rsa]

-l SSH_USER, --ssh-user=SSH_USER

Login account for SSH connections to other Xpand Nodes.

Note: non-root users cannot currently use SSH Hostbased Authentication. [default: current user]

-p SSH_PORT, --ssh-port=SSH_PORT

Port to use for SSH connections to other Xpand Node. [default: 22]

-S, --skip-down-nodes

Skip Xpand Node that are offline or unavailable instead of aborting when one or more Nodes are unresponsive.

-w, --no-passwd

Do not attempt password SSH authentication.

slave

Displays replication slave information.

$ clx [options] slave [slave name]

Command-Specific Option Exceptions:

Option

Description

-n NODE, --node=NODE

Option to run on a specific Xpand Node has no effect for this command.

-X EXCLUDE, --exclude=EXCLUDE

Option to exclude a specific Xpand Node has no effect for this command.

space

Displays storage utilization information for the cluster.

$ clx [options] space

status or stat

Displays general status for the cluster.

$ clx [options] status

Examples:

$ clx --node=1 status
$ clx -n clxdb003 stat
$ clx status

Command-Specific Options

Option

Description

-v, --verbose

Display more details.

CLX Nanny

CLX nanny ensures that all jobs needed for the successful function of Xpand are running.

  • The nanny process is launched automatically by the system init process.

  • The set of jobs to be run by nanny is initially configured via /opt/clustrix/etc/nanny.conf .

  • All jobs for which nanny is responsible are designed to run indefinitely.

  • If any of nanny's monitored jobs stop for any reason, nanny will immediately attempt to restart it.

  • nanny allows stopping and starting of the jobs it controls via the clx utility or the nanny port (2424) .

Jobs Controlled by nanny

The following jobs are kept running by nanny:

Job Name

Description

krobix

Main Xpand process.

  • Located at /opt/clustrix/bin/clxnode

  • Logs to /data/clustrix/log/clustrix.log (location configurable)

Activates:

  • control port (2048)

  • mysql port (3306)

  • health mon port (3581)

statd

Xpand Statistics Collection Script: Regularly collects and aggregates statistics for the UI or other management interfaces.

Actively run on one Xpand Node in the cluster and other Nodes are on standby in the event that the Node fails.

alerter

Monitors cluster and sends email per configuration.

Active on all Xpand Nodes.

clxdbi

Interface required by the Xpand Graphical User Interface (clxgui).

clxgui

XpandGUI Administration UI. Facilitates web-based cluster management.

nanny Commands

To run the nanny command of the clx utility, use the following syntax. The clx utility is located at /opt/clustrix/bin

$ clx [options] nanny nanny_command [job_name]

This is a list of all the nanny commands available.

Command

Description

add_job

Add a new job.

change_job

Change an existing job.

del_job

Delete a configured job.

help

List available commands.

show_job

Display information about a specific job.

show_jobs

List all configured jobs and their status.

start_job

Start a stopped job.

stop_job

Stop a running job.

version

Display the Xpand Node version.