For reporting documentation bugs specifically, see Reporting Documentation Bugs.
MariaDB's bug and feature tracker is found at jira.mariadb.org.
This page contains general guidelines for the community for reporting bugs and feature requests in MariaDB products. If you want to discuss a problem or a new feature with other MariaDB developers, you can find the email lists and forums here.
First, check that the bug or feature request isn't already filed in the MariaDB bugs database.
For the MariaDB issue database, use JIRA search to check if a report you are going to submit already exists. You are not expected to be a JIRA search guru, but please at least make some effort.
Choose Issues => Search for issues;
If the form opens for you with a long blank line at top, press Basic on the right to switch to a simpler mode;
In the Project field, choose the related project, (MDEV for generic MariaDB server and clients);
If you see issue reports which are already closed, pay attention to the 'Fix version/s' field -- it is possible that they were addressed in the upcoming release. If they are said to be addressed in the release that you are currently using or earlier, you can ignore them and file a new one (although please mention in your report that you found them, it might be useful).
If you find an open issue report, please vote/add a comment that the issue also interests you along with any additional information you have that may help us to find and address the issue.
If the issue is not in the MariaDB issue database yet, then it's time to file a report. If you're filing a report about an issue that's already in the , please indicate so at the start of the report. Filing issue reports from MySQL in the MariaDB issue database makes sense, because:
It shows the MariaDB team that there is interest in having this issue addressed in MariaDB.
It allows work to start on addressing the issue in MariaDB - assigning versions, assigning MariaDB developers to the issue, etc.
Bugs and feature requests are reported to the .
Please note that our JIRA entries are public, and JIRA is very good at keeping a record of everything that has been done. What this means is that if you ever include confidential information in the description there will be a log containing it, even after you've deleted it. The only way to get rid of it will be removing the JIRA entry completely.
Attachments in JIRA are also public.
Access to a comment can be restricted to a certain group (e.g. Developers only), but the existing groups are rather wide, so you should not rely on it either.
If you have private information -- SQL fragments, logs, database dumps, etc. -- that you are willing to share with MariaDB team, but not with the entire world, put it into a file, compress if necessary, upload to the , and just mention it in the JIRA description. This way only the MariaDB team will have access to it.
As explained above, all JIRA issues are public. If you believe you have found a security vulnerability, send an email to for, please, do not use JIRA for that. We will enter it in JIRA ourselves, following the practices.
Below is the information we need to be able to fix bugs. The more information we get and the easier we can repeat the bug, the faster it will be fixed.
A good bug report consists of:
The environment (Operating system, hardware and MariaDB version) where the bug happened.
Any related errors or warnings from the server error log file. Normally it is hostname.err file in your database directory, but it can be different depending on the distribution and version; if you cannot find it, run SELECT @@log_error on the running server. If either the variable or the file it points at is empty, the error log most likely goes to your system log. If this is systemd you can get the last 50 lines of the MariaDB log with journalctl -n 50 -u mariadb.service. If possible, attach the full unabridged error log at least from the last server restart and till the end of the log.,
The section below describes which JIRA fields need to be populated while filing reports, and what should be put there. Apart from what's mentioned below, you don't have to fill or change any fields while creating a new bug report.
If you are filing a report for MariaDB server, client programs, or MariaDB Galera cluster, the target project is MDEV. Connectors and MaxScale have separate projects with corresponding names. If you choose a wrong project, bug processing can be delayed, but there is no reason to panic -- we'll correct it. If you inform us about the mistake, we'll change it faster.
Some project names include:
CONC - MariaDB Connector/C
CONCPP - MariaDB Connector/C++
CONJ - MariaDB Connector/J
CONJS - MariaDB Connector/node.js
Feature requests are not the same as bug reports. Specify a New feature type for feature requests in , and a Bug type for bug reports. Like with the project field, choosing a wrong type will put the request to the wrong queue and can delay its processing, but eventually it will be noticed and amended.
See also for things that we are considering to have in the next MariaDB release.
Please make sure the summary line is informative and distinctive. It should always be easy to recognize your report among other similar ones, otherwise a reasonable question arises -- why are they not duplicates?
Examples:
good summary: Server crash with insert statement containing DEFAULT into view
not a good summary: mariadbd crash
Generally, we try not to change the original summary without a good reason to do it, so that you can always recognize your own reports easily.
We do not have separate Severity/Priority fields in JIRA, so this Priority field serves a double purpose. For original reports, it indicates the importance of the problem from the reporter's point of view. The default is 'Major'; there are two lower and two higher values. Please set the value accurately. While we do take it into account during initial processing, increasing the value above reasonable won't do any good, the only effect will be the waste of time while somebody will be trying to understand why a trivial problem got such a high priority. After that, the value will be changed, and the report will be processed in its due time anyway.
Put everything you know about which versions are affected. There are both major versions (10.6, 10.5 etc.) and minor versions (10.5.9, 10.4.12, etc.) available for choosing. Please always specify there the exact version(s) (X.Y.Z) which you are working with, and where you experience the problem.
Additionally, If you know the exact version where the problem appeared, please put it as well. If the problem has been present, as far as you know, in all previous releases, you can also put there the major version, e.g. 10.0. Alternatively, you can mention all of it in the description or comments.
Please also note in the description or comments which versions you know as not affected. This information will help to shorten further processing.
Put here environment-related information that might be important for reproducing or analyzing the problem: operating system, hardware, related 3rd-party applications, compilers, etc.
The most important part of the description are steps to reproduce the problem. See more details about bug report contents above in the section .
If in the process of reproducing, you executed some SQL, don't describe it in words such as "I created a table with text columns and date columns and populated it with some rows" -- instead, whenever possible, put the exact SQL queries that you ran. The same goes for problems that you encountered: instead of saying "it did not work, the query failed, I got an error", always paste the exact output that you received.
Use {noformat}...{noformat} and {code}...{code} blocks for code and console output in the description.
If you have SQL code, a database dump, a log etc. of a reasonable size, attach them to the report (archive them first if necessary). If they are too big, you can upload them to ftp.askmonty.org/private. It is always a good idea to attach your cnf file(s), unless it is absolutely clear from the nature of the report that configuration is irrelevant.
If you found or filed a bug report either in MariaDB or MySQL or Percona bug base which you think is related to yours, you can put them in the Links section; same for any external links to 3rd-party resources which you find important to mention. Alternatively, you can just mention them in the description or comments.
You don't have to set any tags, but if you want to use any for your convenience, feel free to do so. However, please don't put too generic values -- for example, the tag mariadb is meaningless, because everything there is mariadb. Don't be surprised if some tags are removed later during report processing.
Our normal practice is to report a bug upstream if it's applicable to their version. While we can do it on your behalf, it is always better if you do it yourself -- it will be easier for you to track it further.
If the bug affects MySQL, it should also be reported at . If the bug affects Percona server and not MySQL, it should go to .
See the article .
See the article .
If you require personalized assistance, want to ensure that the bug is fixed with high priority, or want someone to login to your server to find out what's wrong, you can always purchase a contract from MariaDB plc or use their consulting services.
This page is licensed: CC BY-SA / Gnu FDL
In the Contains text text field, enter the most significant key words from your future report;
Press Enter or the magnifying glass icon to search.
The content of your my.cnf file or alternatively the output from mariadbd --print-defaults or SHOW VARIABLES.
Any background information you can provide (, tables, table definitions (show-create-table SHOW CREATE TABLE {tablename}), data dumps, query logs).
If the bug is about server producing wrong query results: the actual result (what you are getting), the expected result (what you think should be produced instead), and, unless it is obvious, the reason why you think the current result is wrong.
If the bug about a performance problem, e.g. a certain query is slower on one version than on another, output of EXPLAIN EXTENDED <query> on both servers. If its a SELECT query use .
A test case or some other way to repeat the bug. This should preferably be in plain SQL or in mysqltest format. See mysqltest/README for information about this.
If it's impossible to do a test case, then providing us with a backtrace information would be of great help.
MCOL - ColumnStore
MDBF - MariaDB Foundation Development (anything related to the mariadb.org domain)
MDEV - MariaDB server, client programs, or MariaDB Galera Cluster
MXS - MaxScale
ODBC - MariaDB Connector/ODBC
R2DBC - MariaDB Connector/R2DBC
This page describes how community bug reports are processed among our products and explains what you need to notice while tracking bugs.
MariaDB does not have any SLA or guaranteed reaction times on bugs in Jira. While we are taking bugs reported by the community very seriously, and aim to provide response and to handle issues as fast as possible, MariaDB does not have a dedicated bug verification team, this activity is performed on the best-effort basis.
To make sure your bug report will be confirmed and moved forward faster, please follow about creating bug reports.
As of today, initial bug processing routine in MariaDB is not strictly formalized. This section describes the de-facto status rather than any policy.
The process is different for bug reports (Bug type in JIRA) vs feature requests (Task type). The process described below is related to bug reports.
All new bug reports go to the waiting list, to be reproduced and confirmed by a member of the team. The bug stays in the queue until one or more of the conditions below are met:
Bug report is assigned to a developer;
Bug report gets status 'Confirmed';
Bug report gets the label 'upstream';
Bug report is closed (on whatever reason).
With other things equal, bug reports in the queue are initially handled in the FIFO manner; however, there are various factors that make things not equal.
First thing that is taken into account is Priority. It does not mean that everything needs to be filed as Critical; on the contrary, it means that Priority should be chosen wisely. Although a report with higher Priority will be looked at sooner, as soon as it becomes clear that the Priority is set to a higher value than the problem deserves, it will be amended and put back to the queue. However, if the high priority is justified, we will try to process the report as fast as possible.
Another important factor is the quality of the report itself.
If the report is written well and has all information, including a reproducible test case, it can be verified and moved forward quickly.
If the report is written clearly enough, but does not have enough information, it will get fast enough first response where we will request the missing details; but the further exchange can take a lot of time until we get everything we need to work on the issue.
Finally, reports which are written in a tangled and incomprehensible manner get the slowest response time, because even if eventually it turns out that they do have all required information, it is difficult and time-consuming to extract and process, so they can be put aside for some time.
Complete processing of a reported bug can be complicated and time-consuming, especially the reproducing part. We do not want our users to wait for long not knowing if their bug report has even been noticed, we try to provide first response quicker than that.
First response to the bug, which we are trying to provide as quickly as possible, is one of these:
If we can reproduce the problem based on the information that was provided in the initial description, the report gets the status Confirmed.
If it is obvious from the initial description that the bug report is a duplicate of an existing one, or the problem has already been fixed in later releases or in the upcoming release, or the described behavior is not a bug, or, in very rare cases, it is admitted to be a bug, but it is not going to be fixed, the report gets closed with the corresponding Resolution value and a comment with the explanation.
If the bug report at least appears to describe a real bug, but we do not have enough information to proceed, we will request the information from the reporter, and the report will go to the Need feedback list.
If on some reason it is clear from the bug report that it will be very difficult to reproduce based on the information from the user, but there is a reason to believe that the problem can be analyzed by code inspection, the bug report can be assigned to a developer who is an expert in the corresponding area for the analysis.
We realize that "as quickly as possible" is a relative term. The dream scenario is that all reports are responded to in a matter of hours; however, more realistically, it can take a few days, and in some cases, when the team is overly busy with a big upcoming release or some other extraordinary events, it can even be weeks.
When a report does not have all the information to reproduce the problem right away (which is quite often the case), we will ask the reporter to provide the missing information. Usually it takes more than one iteration to get everything right, so it is important that you respond to the questions as precisely as you can. Please make sure that you answered all questions (or, if you cannot answer some of them, please say so, otherwise we will have to ask again, and more time will be wasted on it).
There is no status "Need Feedback" in our JIRA; instead, we are using the label need_feedback. As long as the report has this label, it remains on the "Waiting for feedback" list. The label is set and removed manually by whoever asks for the feedback and receives it; so it can happen that the reporter has provided the response, but it remained unnoticed and the bug keeps waiting. It will be our fault, but human errors happen; it would help a lot if the reporter removed the label along with providing the feedback.
This question arises fairly often, so it deserves mentioning.
As already said before, the need_feedback label is set and removed manually. JIRA e-mail updates about it can be confusing when you look at them quickly. For example, when someone removes the label, the email looks like this:
What it says that the Labels field has become empty, while before it had been need_feedback. People often misread it and ask "What else do you need from me? I've answered your questions". This update means that at the moment we don't need anything, your report is back to the incoming queue, and your feedback will be analyzed as soon as possible. Then, we will possibly ask more questions and set the label again, and the notification will look like this:
If the feedback exchange was fruitful and we received enough information to proceed, the bug report will go through the normal verification steps.
Reports do not stay open on the "Need Feedback" list forever. After a month of waiting, if we do not get a response from the reporter, and still cannot proceed without it, we close the report as Incomplete with the corresponding comment. This state is not irreversible: you can still add comments and provide the information even when the report is closed as Incomplete, and it will be re-opened.
Sometimes it happens that after iterations of feedback requests we run out of ideas what else to ask from the reporter, and still could not verify the bug, or that the reporter is willing to collaborate with us, but cannot provide the necessary information on objective reasons (usually when the problem happens on a production instance). In some cases we might close the report as "Cannot reproduce", which we consider our loss; but more often we want to keep it open, in hope that more information arrives, maybe from a different source, and together with this report they will help us get to the bottom of the problem; if it happens so, the report gets assigned to somebody without being confirmed, just so it remains at least on somebody's radar, and it will stay open for a long time. It does not mean it is forgotten, it means that for the time being we hit the wall. You are very welcome to comment on such reports, whenever you think you might have something to add, because this is exactly what we are waiting for.
Normally the bug report has to go through the following steps before it is moved forward to fixing:
the described problem needs to be reproduced;
it needs to be checked against all active post-Beta versions of MariaDB where it is theoretically applicable (as of the moment of writing this article, it is 5.5, 10.0, 10.1);
in case it is a relatively recent regression, the guilty change needs to be found;
the component or functional area should be determined, so that the bug gets assigned to the right person.
After that the bug is ready for fixing.
Sometimes it seems hard to understand from the outside how MariaDB development team chooses which bugs to fix in a particular release, or why some bugs are fixed faster than others, or why critical bugs stay untouched for a long time.
MariaDB currently uses 1- or 2-week sprint model for server development and bugfixing. It needs a separate article to describe it in more detail, but for bugfixing, in a nutshell it means the following.
one or two weeks before a scheduled release the team creates a new sprint and evaluates existing bugs which affect this release;
the selected bugs are added to the new sprint;
during the active sprint, the developer is supposed to work on the tasks which are part of the sprint, unless a true emergency arises.
There are two important consequences of this model which sometimes cause a confusion:
If the current sprint is for one version, e.g. 10.0, and you file a bug for another version, e.g. 10.1, then, even if the bug is really critical, it won't be jumped on right away: it makes no sense, because the 10.1 is not going to be released next week anyway, while 10.0 will be. When the 10.0 sprint finishes, and 10.1 sprint starts, your bug will be picked up for that sprint and fixed then.
If the current sprint for 10.1 is already in progress, newly created 10.1 reports normally won't be included into it, unless they are considered so extremely important that the developer is allowed to ignore the sprint plan.
When a new sprint is created, bugs which affect the scheduled release are evaluated.
from all such bugs assigned to a developer, each developer chooses bugs he is able to work on during the given time interval;
bug priority plays the most significant role in this process, but this is not the only factor.
Blocker bugs must be either fixed or degraded before the release goes out;
Critical bugs should be chosen above other bugs, except for Blockers;
among Major bugs,
bugs with patches, either external, or upstream, or internal, are usually prioritized above ordinary bug reports;
external reports (community reports) are ranked higher than bugs reported by the development team;
bugs which can realistically be fixed in the given time interval are chosen more frequently than those that are likelly to take several cycles;
Minor bugs are usually fixed when there are no more urgent tasks.
If a bug report has passed through verification stage, either being confirmed, or pushed forward to the development-level analysis as is, there can be various updates on it. It is important to understand what they mean.
All JIRA fields are public, but some of them are mainly used for internal development process, while others are more user-facing. This article describes which fields should be populated during the initial report submission. There is a different set of fields important for tracking purposes.
It might come as counter-intuitive, but in the existing JIRA structure, the Status field does not mean much for the user, it is mainly used for development and management purposes. On the contrary, the Resoluton field is entirely user-facing: it does not participate in planning or development. It remains the same 'Unresolved' for the whole life of the report, and is only changed when the bug gets closed, demonstrating the reason why it was closed.
Resolution
Unresolved - the bug report remains open, the work has not been finished.
Fixed - the bug has been fixed, see Fix version/s and possibly comments to the report for more information. This is almost always a terminal state, we do not re-open fixed bugs even if they later re-appear; please create a new one instead. The only case when it can be re-opened is when the 'Fix version/s' have not been released yet.
Duplicate - the bug report is identical to an already existing open (or recently fixed) report, which will be quoted in the comments and/or links. It is usually a terminal state, unless it is proven later that the report was not a duplicate after all.
Not a bug - the described behavior is not a bug, there will be a comment explaining why. It is usually a terminal state, unless you object and show why it is a bug. If the report is in fact a feature request, then rather than closing it as 'Not a bug', we will switch the type to 'Task'.
Incomplete - we had requested feedback from the user and waited for 1 month, but did not receive it. It is a pseudo-terminal state, the report can be re-opened any time when the requested information is provided.
Cannot reproduce - rather rarely used "resolution", which means we could not find the way to confirm the problem described by the reporter, and ran out of ideas what other information to request from the reporter in order to reproduce it.
Won't fix - another rarely used "resolution", which means that the bug is admitted, but we have no intention to fix it. Usually it happens when the bug only affects old versions, and is not important enough to fix in the old versions; or, when it is related to systems or architectures we don't officially support.
Status
Open, Confirmed - this distinction is used in our internal queues, but from the user's perspective the difference is slim: setting the bug report to 'Confirmed' does mean that we are satisfied with the information provided in the report, but the user will also know about it from our comments and other updates. Otherwise, bugs in both statuses can be considered for fixing.
In Progress, Stalled - different intermediate states of bugs which help developers to filter their lists and management to gather a picture of the current activity. For the user, there is no important difference -- despite the negative semantics, 'Stalled' does not mean that something is wrong with the bug report, only that the developer is not working on it actively at the moment.
In review - means, literally, that a peer review has been requested.
Closed - means that the bug report is closed, on whatever reason. The real reason is in the 'Resolution' field.
This is an important field for progress tracking.
After the bug is confirmed or otherwise acknowledged, this field is populated with a set of major versions where we intend to fix it. E.g. if the field is set to 10.0 10.1, it means that at the moment we consider it for fixing in some future 10.0 release (not necessarily the next one), and the bugfix will be merged into the next 10.1 release after that; but we do not consider it for fixing in 5.5, even if it is affected to.
To some extent, you can influence the initial plans: if you see that the fix is not targeted for versions where you think it should be, you can comment on the report, and if you provide convincing arguments and make your case, it can be reconsidered.
The value of the field is not a promise to fix the bug in the mentioned releases. It can be changed both ways: during further analysis, the developer can find out that it can be safely fixed in an earlier release, or, on the contrary, that it cannot be safely fixed in the GA release, and the fix can only go to the next versions which are currently under development.
After the bug is fixed, the value of the field is changed to the exact versions, e.g. 10.0.25 10.1.14. It means that the patch has been pushed into the 10.0 branch, and will be released with 10.0.25 release; it also means that the patch will be merged to 10.1 tree and released with 10.1.14 release, but it does not mean that it is already in the 10.1 branch.
As the other article says, the Priority field serves two purposes. During the initial bug creation, it indicates the importance of the bug report from the user's perspective (in other bug tracking systems it is called 'Severity' or alike). After the bug has been confirmed, the same field is used for development purposes, to prioritize bug fixing (real 'Priority'). While we take into account the reporter's view on the matter, we can change the initial priority both ways, depending on the information we revealed during the problem analysis, versions affected, etc.
The value of the field normally means the following:
Blocker - we currently think that the bug must be fixed before the next release(s) set in the 'Fix version/s' field;
Critical - the bug should be picked up for fixing earlier than any other bugs apart from blockers;
Major - the bug will be present in the main queue for fixing in the upcoming 'Fix version/s', although only a part of such bugs will be fixed in every release;
Minor, Trivial - the bugs will be picked up when the assignee does not have more pressing issues for the upcoming release.
Please note that the Priority field only demonstrates our intentions at the moment, it does not guarantee that things will happen according to these intentions.
Labels are mostly used for more convenient filtering and don't carry much importance otherwise. However, there are a few that affect the processing of a bug report:
need_feedback - its role during the initial bug processing was already described above. However, after a bug is confirmed and queued for fixing, it should not appear anymore; and even if it's left by mistake, it won't affect the progress.
upstream - the label means that the bug also exists in the upstream version of the corresponding component - normally, in MySQL server or a client program, but can also be in Percona's XtraDB or TokuDB. Normally there should also be a link to the upstream bug report. Setting this label means that we might want to take for a while and see whether the bug is fixed in the upstream version before we fix it in MariaDB directly. It was usual for 5.5, less usual for 10.x where bugfixes, apart from InnoDB, are not merged automatically. The label is still set, but it is more for informational purposes than to affect the priority.
upstream-fixed - the label means that the bug used to exist in the upstream version, but not anymore. It means that there is nothing more to wait; moreover, it might be worth picking up the bug soon and at least evaluating the upstream bugfix.
MariaDB encourages contributors to provide bug fixes; so, bug reports which come with the fixes in general have a quicker turnaround. The bug fix can come in a form of Git pull request, or, in simple cases, as a diff pasted in or attached to the bug report itself.
There are some basic rules for bugs, particularly for setting the Resolution value, which we want to stick to and which might be different from procedures you came across in other projects. It mainly concerns external bugs (those that come from the community), for internal ones we can cut corners more freely.
This all is easier to understand if one remembers that the Resolution or its analogues in other bug-tracking systems is a user-facing field, as already mentioned above, and that it relates more to the report, than to the bug itself.
An older bug report cannot be a duplicate of a newer one, it is nonsensical. The only possible exception is when an older bug has no useful information whatsoever and the reporter does not provide any helpful feedback, while a newer report was not closed as a duplicate right away and got some useful updates. The common example of such exception is when the first report is just an optimized stack trace, no query, no data, nothing to work with, while the second report has a test case. But if the first reporter at least makes an effort to collaborate, the report deserves to be treated with respect.
Bug reports which have essentially different descriptions and/or test cases should not be duplicates. The common example is this: a developer creates a bug saying something like "this and that pieces of code are wrong, it should be so and so"; and then a user files a bug saying "this SQL produces a wrong result on this data set". Even if they are about the same error in the code at the end, they are not duplicate bug reports.
Obviously, a report can never be a duplicate of anything private (luckily it does not concern MariaDB server so far, as the bug reports are public).
In general, a bug report is a duplicate of another one if, and only if, the new reporter could find the existing report just by a reasonable JIRA search.
A bug report should not be closed as "cannot reproduce" if it was once verified/confirmed, but disappeared in later versions. It's unfair to the reporter, and also dangerous to the product. We should know why a bug stopped being reproducible -- either we find when and how it was fixed (and close the report as "Fixed in version X by a patch for Y"), or we discover that it wasn't in fact fixed, but just masked. The simplest example is a change of execution plan in optimizer: server would crash on a particular query, then due to a change in optimizer it started using a different plan for the same query, so it wouldn't go through the crashing path anymore. The crash is still there, though.
In general, the "cannot reproduce" resolution is a last resort. Usually if we can't reproduce something, it means that either the reporter did not provide required information (and then the resolution should be "Incomplete"), or we don't know what to request from the reporter, and then we should keep thinking, rather than close it. Of course, it happens that the bug is genuinely not reproducible, but it shouldn't be decided lightly.
This page is licensed: CC BY-SA / Gnu FDL
How and where the community can report bugs and file feature requests, and how bug reports are processed for community users. MariaDB plc provides SLA for customer issues (see ).
To report documentation issues, please review .
Elena Stepanova updated MDEV-9791:
----------------------------------
Labels: (was: need_feedback)Elena Stepanova updated MDEV-9801:
----------------------------------
Labels: need_feedback (was: )bugs which affect the reporter in a worse matter get more attention than those that have viable workarounds;
Instructions on how to build a mysqld that contains all the information we need to fix problems you encounter. (A more detailed explanation can be found .)
Add the option to your /.my.cnf or /etc/my.cnf file under the [mysqld] tag.
Get the latest .
Compile MariaDB with the -g compiler flag (Unix).
Optionally: with more checking Compile MariaDB for debugging - will cause slowdown.
Shut down your old mysqld server.
Install the new compiled mysqld binary. Note that if you are compiling same version of MariaDB that you have already installed it's enough to just copy this one binary!
Restart mysqld.
Compiling with -g should not cause any notable slowdown of the server.
You can of course also do make install, but the above way allows you to go back to your old binary if needed.
If you get any errors about a wrong number of error messages, you can fix that by copying the corresponding language file from sql/share over your old ones (this should be reasonably safe to do).
Now when you get a crash do the following:
Create a README file that describes the problem. You can use the mysqlbug script to generate a template for this.
Create a tar file containing the core, the mysqld binary and README. If possible, also add any database files that could help us repeat the problem!
Send it to our secure ftp server:
To be able to follow the progress, create a bug report in JIRA about this. This should be easy to do based on the information you have in your README file.
This page is licensed: CC BY-SA / Gnu FDL
Instructions to narrow down problems on a replication replica.
Sometimes a event causes an error of some sort. A whole binary log file is sometimes impractical due to size or sensitivity reasons.
Step 1: Copy the binary log locally
This is just in case you don't quite extract the right information first. If the binlog expired off and you haven't got the right information, your bug report may not easily be reproducible.
Step 2: Create an extract header
Binary logs have a header portion. Without the header won't be able to read it. The header also contains valuable session information
We look at the binary log to see how big the header and session information is:
We see that the session information ends at 328 because of the last line, so we extract to that point.
We need to find out at what offset the entry at 129619 ends and it might be useful to extract some previous entries as well.
cp sql/share/english/* mariadb-install-dir/share/mysql/englishsh> tar cvfz /tmp/mariadb-bug-'short-description'.tgz mariadb-data-dir/core* mariadb-install-dir/libexec/mysqld READMEsh> ftp -a ftp.askmonty.org
ftp> cd private
ftp> binary
ftp> put /tmp/mariadb-bug-'short-description'.tgz
ftp> quitNow let's assume we want to start at our original 129619 and finish before 130168
Check the extract:
Upload this to the private uploads or attach to the public bug report if nothing sensitive there.
This page is licensed: CC BY-SA / Gnu FDL
sudo cp /var/lib/mysql/mysql-bin.000687 ~/
sudo chown $USER: ~/mysql-bin.000687mariadb-binlog --base64-output=decode-rows --verbose mysql-bin.000687 | more
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#150323 22:45:58 server id 76 end_log_pos 245 Start: binlog v 4, server v 5.5.39-MariaDB-log created 150323 22:45:58
# at 245
#150323 22:45:58 server id 76 end_log_pos 328 Query thread_id=9709067 exec_time=0 error_code=0
SET TIMESTAMP=1427116558.923924/*!*/;
SET @@session.pseudo_thread_id=9709067/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=0/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=8/*!*/;
SET @@session.time_zone='SYSTEM'/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
BEGIN
/*!*/;
# at 328dd if=mysql-bin.000687 of=mysql-bin.000687-extract-offset-129619 bs=1 count=328mariadb-binlog --base64-output=decode-rows --verbose mysql-bin.000687 | grep '^# at ' | grep -C 10 '^# at 129619$'
# at 127602
# at 127690
# at 128201
# at 128290
# at 128378
# at 128829
# at 128918
# at 129006
# at 129459
# at 129548
# at 129619
# at 129647
# at 130070
# at 130097
# at 130168
# at 130196
# at 130738
# at 130942
# at 130969
# at 131040
# at 131244mariadb-binlog --base64-output=decode-rows --verbose --start-position 129006 --stop-position 130168 mysql-bin.000687 | moredd if=mysql-bin.000687 bs=1 skip=129619 count=$(( 130168 - 129619 )) >> mysql-bin.000687-extract-offset-129619mariadb-binlog mysql-bin.000687-extract-offset-129619Profiling the memory usage can be useful for finding out why a program appears to use more memory than it should. It is especially helpful for analyzing OOM situations or other cases where the memory grows linearly and causes problems.
To profile the memory usage of a program, there are multiple options. The following sections describe the methods that are available.
If a problem in memory usage is identified and it appears to be due to a bug, please open a new bug report on the MariaDB Jira under the correct project and include the relevant memory profiling output in it. Refer to How to Write a Good Bug Report for more details.
Transparent huge pages (THP), which is enabled by default in many newer Linux distributions, can cause out-of-memory-issues for MariaDB as THP is not suitable for databases. This is described at .
MariaDB Community Server 10.6.17, MariaDB Enterprise Server 10.6.16-11 and all other MariaDB server releases after these have THP disabled.
If and do not show an increase in memory, but the process still increases in size, then a likely problem is the system memory allocation library (malloc). Replacing malloc with should fix the issue in this case.
Recent MariaDB versions have a global variable that shows how much memory the MariaDB server has allocated. By monitoring this variable one can find out if if the MariaDB allocated memory grows.
One can also check memory usage per user with the :
This shows the current memory used per connection and the maximum memory they have used since the user connected.
The can also be used to find out who is allocated memory and for what.
Note that one can also set the variable to restrict a user's memory usage.
The toolkit comes with the memleak program that traces outstanding memory allocations. This is a very convenient way of debugging high memory usage as it'll immediately show where the memory is allocated at.
By default the tool will print output once every five seconds with the stacktraces that have the most open allocations. Ctrl+C can be used to interrupt the collection of the traces.
The profiling interval and the profiling duration can be passed as arguments to memleak. The first argument is how often a sample is taken and the second argument is how long to sample for. To help analyze excessive memory usage, collect the output of the memleak program for at least 60 seconds. The longer the profiling can be left on, the more accurate the information will be.
The overhead of the profiling can be large enough that it affects production workloads negatively. To reduce the overhead, the sampling frequency of memory allocations can be lowered using the --sample-rate option:
For example, -s 10 will sample only 10% of memory allocations which may miss out memory leaks from individual allocations but the longer the system is left running, the more likely it is that a leaking memory allocation is sampled. This means that even with a lower sampling rate, the source of the memory leak will eventually be found.
On RHEL based systems, the package is named bcc-tools. After installing it, use the following command to profile the memory usage 5 times per second over a window of 60 seconds:
On Ubuntu/Debian the package is named bpfcc-tools. After installing it, use the following command to profile the memory usage 5 times per second over a window of 60 seconds:
Jemalloc is an alternative to the default glibc memory allocator. It is capable of analyzing the heap memory usage of a process which allows it to be used to detect all sorts of memory usage problems with a lower overhead compared to tools like Valgrind. Unlike the ASAN and LSAN sanitizers, it is capable of detecting cases where memory doesn't actually leak but keeps growing with no upper limit (e.g. items get appended to a list but are never removed).
To enable jemalloc, the packages for it must be first installed from the system repositories. Ubuntu 20.04 requires the following packages to be installed for jemalloc profiling:
The version of jemalloc that is available in most Red Hat repositories is not compiled with memory profiling support enabled. For RHEL based distributions, the only option is to .
Once installed, edit the systemd service file with systemctl edit mariadb.service and add the following lines into it. The path to the libjemalloc.so file is OS-specific so make sure it points to the correct file. The example here is for Ubuntu and Debian environments.
Then create the directory for the profile files:
And finally restart MariaDB with systemctl restart mariadb.service.
The directory in /var/lib/mysql/jeprof/ will start to be filled by versioned files with a .heap suffix. Every time the virtual memory usage reaches a new high, a file will be created. Initially, the files will be created very often but eventually the pace will slow down. Once the problematic memory usage has been identified, the latest .heap file can be analyzed with the jeprof program.
The simplest method is to generate a text report with the following command.
A better way to look at the generated heap profile is with the PDF output. However, this requires the installation of extra packages (apt -y install graphviz ghostscript gv). To generate the PDF report of the latest heap dump, run the following command:
The generated heap-report.pdf will contain a breakdown of the memory usage.
Note that the report generation with the jeprof program must be done on the same system where the profiling was done. If done elsewhere, the binaries do not necessarily match and can cause the report generation to fail.
Similarly to the jemalloc memory allocator, the memory allocator comes with a leak checker and heap profiler.
On RHEL based systems, the gperftools package is in the EPEL repositories. These must be first enabled by installing the epel-release package.
After this, the gperftools package can be installed.
Once tcmalloc is installed, edit the systemd service file with systemctl edit mariadb.service and add the following lines into it.
Note: Make sure to use the correct path and library name to the tcmalloc library in LD_PRELOAD. The following example uses the Debian location of the library. The file is usually located in /usr/lib64/libtcmalloc_and_profiler.so.4 on RHEL systems. The version number of the library can also change which might require other adjustments to the library path.
Then create the directory for the profile files:
And finally restart MariaDB with systemctl restart mariadb.service.
The heap profiling is configured using environment variables. The details can be found in the Modifying Runtime Behavior section of the gperftools documentation:
By default, tcmalloc dumps the heap profile every time 1GiB of memory has been allocated (HEAP_PROFILE_ALLOCATION_INTERVAL) or whenever the high-water memory usage mark increases by 100MiB (HEAP_PROFILE_INUSE_INTERVAL). If there's no activity, no memory dumps will be generated.
To trigger a memory dump based on a time interval, set the HEAP_PROFILE_TIME_INTERVAL environment variable to the number of seconds between each dump. For example, with Environment=HEAP_PROFILE_TIME_INTERVAL=3600 there will be one heap dump per hour.
Depending on which OS you are using, the report generation program is named either pprof (RHEL) or google-pprof (Debian/Ubuntu).
It is important to pick the latest .heap file to analyze. The following command generates the heap-report.pdf from the latest heap dump. The file will show the breakdown of the memory usage.
This page is licensed: CC BY-SA / Gnu FDL
SELECT id, MEMORY_USED, MAX_MEMORY_USED FROM information_schema.processlist;-s SAMPLE_RATE, --sample-rate SAMPLE_RATE
sample every N-th allocation to decrease the overheadsudo /usr/share/bcc/tools/memleak -p $(pidof mariadbd) 5 60 | tee memleak.logsudo memleak-bpfcc -p $(pidof mariadbd) 5 60 | tee memleak.logapt-get -y install libjemalloc2 libjemalloc-dev binutils[Service]
Environment=MALLOC_CONF=prof:true,prof_leak:true,prof_gdump:true,lg_prof_sample:18,prof_prefix:/var/lib/mysql/jeprof/jeprof
Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2mkdir /var/lib/mysql/jeprof/
chown mysql:mysql /var/lib/mysql/jeprof/jeprof --txt /usr/sbin/mariadbd $(ls -1 /var/lib/mysql/jeprof/*.heap|sort -V|tail -n 1) > heap-report.txtjeprof --pdf /usr/sbin/mariadbd $(ls -1 /var/lib/mysql/jeprof/*.heap|sort -V|tail -n 1) > heap-report.pdfsudo dnf -y install epel-releasesudo dnf -y install gperftoolssudo apt -y install google-perftools[Service]
Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_and_profiler.so.4
Environment=HEAPPROFILE=/var/lib/mysql/pprof/mariadbd.prof
Environment=HEAPCHECK=normal
Environment=HEAP_CHECK_AFTER_DESTRUCTORS=truemkdir /var/lib/mysql/pprof/
chown mysql:mysql /var/lib/mysql/pprof/pprof --pdf /usr/sbin/mariadbd $(ls /var/lib/mysql/pprof/*.heap|sort -V|tail -n 1) > heap-report.pdf