Databases news

SpyMemcached Transcoder with PHP PDO Objects using ZLIB

Mon, 06 Feb 2012 22:40:00 GMT

My technology stack services more then 2 Million Daily Active users. Its very basic. PHP talks to mySQL, Memcache, RabbitMQ, Gearman and Facebook. Now that we have more Java specifically to support our SmartFox Server and other services, blurring the lines between what data is set in PHP and what data is read in Java is very necessary.Java J-Connect makes reading mySQL Data as simple IMHO as PHP's PDO. What is hard is reading PHP's serialized format from PHP's Memcache library.In PHP there are two main C backed Libraries. There is Memcache the original PHP library which I happen to use, and Memcached which is the library I wanted to use but didn't deploy since EC2 package system conflicted and cause issues (I fixed them but to late to deploy). Memcache stores data in PHP's serialized format and compresses it via ZLIB, while Memcached can store data as PHP's serialized format, JSON, Binary Serialized (which is rather awesome), JSON Array Notation and has a multitude of compressing formats none of which are pure ZLIB that I noticed.Here is the problem. Spymemcached is a lib for talking to memcache but can't unserialized PHP serialized format (or read it natively and return a string) and cannot Decompress ZLIB but can Decompress GZIP. Now a great speed up would be to use PHP's serialized data set from PHP and share memcache resources from PHP and Java just like what is done for the mySQL resources.What needs to be done? Well, build your own Transcoder for Spymemcached. Fortunately Spymemcached documented an interface to do just that.What is needed. Implement the spymemcached Interface defined here. Use org.lorecraft.phparser to unserialize PHP data defined here. Return the Object. Below is the code.package com.schoolfeed.spymemcached;import java.io.ByteArrayInputStream;import java.io.ByteArrayOutputStream;import java.io.IOException;import java.util.zip.InflaterInputStream;import net.spy.memcached.CachedData;import net.spy.memcached.compat.CloseUtil;import net.spy.memcached.transcoders.BaseSerializingTranscoder;import net.spy.memcached.transcoders.Transcoder;import org.lorecraft.phparser.*;public class PHPSerializedTranscoder extends BaseSerializingTranscoder implements Transcoder { static final int COMPRESSED=2; /** * Get a serializing transcoder with the default max data size. */ public PHPSerializedTranscoder() { this(CachedData.MAX_SIZE); } /** * Get a serializing transcoder that specifies the max data size. */ public PHPSerializedTranscoder(int max) { super(max); } /** * decode the byte data from Memcache decompress it if necessary and return the Object * @param CacheData - the byte data is turned into a object * @returns Object */ public Object decode(CachedData d){ byte[] data=d.getData(); Object rv=null; String ds="N;"; if((d.getFlags() & COMPRESSED) != 0) { getLogger().debug("Looks like d is compressed"); data=decompress(d.getData()); } ds=decodeString(data); getLogger().debug("DECODED: [" + ds + "] about to SerializedPhpParser"); SerializedPhpParser sp = new SerializedPhpParser(ds); try { rv = sp.parse(); getLogger().debug("Parse was cool!!"); } catch(Exception e){ getLogger().debug("Not a PHP Object? : " + ds); rv = ds; } return rv; } /** * PHP Memcache stores compress data in ZLIB format override the base class decompress method to handle ZLIB * * @param byte array - raw data from Memcache * @returns byte array */ protected byte[] decompress(byte[] in) { ByteArrayOutputStream bos=null; final int BUFFER = 2048; if(in != null) { ByteArrayInputStream bis=new ByteArrayInputStream(in); bos=new ByteArrayOutputStream(); InflaterInputStream iis = null; try { iis = new InflaterInputStream(bis); byte[] buf=new byte[BUFFER]; int r=-1; while((r=iis.read(buf, 0, BUFFER)) > 0) { bos.write(buf, 0, r); } } catch (IOException e) { getLogger().warn("Failed to decompress data", e); bos = null; } finally { CloseUtil.close(iis); CloseUtil.close(bis); CloseUtil.close(bos); } } return bos == null ? null : bos.toByteArray(); } /** * encode -- not implemented yet * */ public CachedData encode(Object o){ int flags = 0; byte[] b=null; return new CachedData(flags, b, getMaxSize()); } /** * no need to async Decode let's do it realtime */ public boolean asyncDecode(CachedData d) { return false; }}This is a stop-gap solution until we make the transition to Memcached with JSON encoding. Then I can use Jackson-JSON - which is a fast JSON encoder/decoder for Java enabling a great portable message protocol between the two stacks and nearly any other language we might add to the system (like Python).

Dot-Org Pavilion at the Percona Live MySQL Conference

Mon, 06 Feb 2012 22:38:45 GMT

Are you involved with an open-source project that’s interesting to MySQL users, such as Nginx, PHPMyAdmin, Drupal, Jenkins, PHP, and so on? Percona just published the application form for dot-org groups to have a free expo hall booth in the Percona Live MySQL Conference in April. Please submit your applications now, and tell your friends about this, because a) the schedule for applying is very short, and b) space is limited. For those of you who don’t know what this is, it’s another of the O’Reilly traditions we’re trying to continue. (We are trying very hard to make this event as close to a clone of O’Reilly’s as we can.) It’s a free table in the expo hall where people who participate in a non-commercial open source project can exhibit. I organized a Maatkit booth a few times in the past, and was always really grateful to O’Reilly for making the space available. Space in the expo hall is at a premium, but we think that these dot-org booths are even more valuable to the open-source projects and the conference attendees. So, please tell your friends who care about open source, and ask them to tell their friends too. Let’s get some great open-source projects into the expo hall, alongside the commercial vendors! Further Reading:Percona Live MySQL Conference comes to Washington DC One-day Percona Live conference in San Francisco Maatkit at the dot-org pavilion at MySQL Conference and Expo 2009 Schedule for MySQL-and-beyond conference is live Percona Live from a community insider’s perspective

MySQL progress in a year

Mon, 06 Feb 2012 22:22:21 GMT

Usually people do this around New Year, I will do it in February. Actually, I was inspired to do this after reviewing all the talks for this year's MySQL Conference - what a snapshot into the state of where we are! It made me realize we've made important progress in the past year, worth taking a moment to celebrate it. So here we go... Diversification read more

Upcoming talks at PHP UK, Confoo and your event

Mon, 06 Feb 2012 21:25:54 GMT

I will be speaking on new MySQL features at the PHP UK conference on February 25th in London. A few days later on march 3rd, I will be presenting the Care and Feeding of a MySQL Database at Confoo in Montreal. If you are attending either event, please introduce yourself! Fellow Community Manager Keith Larson and I working hard to keep up with all the upcoming shows. If you think we might have not notices your upcoming event, please drop us a line. We will have thumb drive, baggage tags, stickers, and maybe (fingers crossed) t-shirts or plush dolphins soon.

New England’s Victory (for Big Data)

Mon, 06 Feb 2012 17:06:01 GMT

While it might not have been New England’s weekend on the Big Gridiron, it was certainly New England’s day for Big Data at the New England Database Summit on Friday at MIT. The summit was well attended, with 350 registrants and keynotes from prominent MySQL users such as Mark Callaghan. The coverage was quite broad, with presentations running the gamut from grad students (complete with bodyguards and intimidating academic advisors) to established companies such as StreamBase. The sponsor list was an A-list this year as well, with EMC and Microsoft being the two biggest backers. There were far too many and diverse topics to write about all of them. That said, here were a few of the notable ones. Keynote #1: Johannes Gehrke (Cornell): Declarative Data-Driven Coordination Johannes Gehrke of Cornell kicked off with the first keynote on Declarative-Driven Coordination. His methodology shed light on an alternative to out-of-band communication. The presentation focused on how to successfully handle entangled queries. More Sleep for Tom and Meg if They Can Just Coordinate In brief, what he showed is a way for someone to see if their friend is on a flight and have the database go about satisfying mutual constraints. With a proof that is outlined in his Sigmod paper, his main theorem is that any schedule that is entangled-isolated is also oracle-serializable. It’s a clever approach, as long as one’s set of friends being entangled remains small. Keynote #2: Mark Callaghan (Facebook): Performance is Overrated The room got a little quiet when Mark took the stage. Some people were expecting a possible rehash of this summer’s brouhaha between Mike Stonebraker and Facebook on the fate of MySQL. But, instead Mark jumped into some very practical discussions about managing MySQL at scale. First, he noted that manageability needs more attention since… The cost of extra hardware can be predicted The cost of downtime cannot Downtime comes in many forms (server down and server too busy) For Mark, manageability has a number of meanings. This includes the rate of interrupts/server for the operations team. Mark finds that while the server count grows quickly, his operations team grows slowly. Hence, it is imperative that the quality-of-service improve over time (i.e., Does work get done? Does work get done on time?). Mark and his team use MySQL for a number of reasons. First, it was there when Mark arrived. Second, Mark and his team made it scale 10x. Finally, Mark likes MySQL for OLTP. As Facebook has grown though, so have the number of servers. This is due to “Big Data” x high QPS. Hence, they have had to add servers to add IOPs. To address this, Mark noted that flash memory (SSD) is very interesting as are (we blush) write-optimized databases. The last part of his presentation focused on advice for scaling: More Data, More QPS. His tips were quite straightforward: Fix stalls to make use of capacity Don’t make MySQL faster, make it less slow Improve efficiency to use less Repeat Additional details can be found in Sheeri’s excellent live blog of the presentation. New Tools and Systems Session: Willis Lang (University of Wisconsin): Energy-Conscious Data Management Systems Just as Mark stressed that performance isn’t everything when he spoke about management, Willis Lang pointed out another key concern. His slides noted that “three decades of database research has optimized for the highest possible performance possible regardless of energy consumption.” (We agree and have written about this topic as well). Willis and his team have been looking at various techniques for addressing this such as using variable speed disks. He has been systematically studying the power/performance trade-offs of hardware components. The preliminary memory-based results showed that interesting trade-off opportunities exist if one rethinks database design principles. His presentation focused on the improvements that can be seen with memory parking. Additional details on his research can be found here. As mentioned previously, there were many good talks — much more could be written about the event. Other interesting speakers included David Karger who introduced Dido, which seeks to make database manipulation as easy as document editing, and Alvin Cheung whose Pyxis project eases application development with automatic code partitioning based on application and server characteristics. Kudos to Samuel Madden (MIT) and Ugur Cetintemel (Brown University) for organizing the event. Additional details can also be found via the Twitter hashtag #nedb12 and the event homepage.

Building XtraDB on Windows

Mon, 06 Feb 2012 16:19:41 GMT

As you may or may not know, Windows is not yet a supported platform for XtraDB. I thought I’d try to build it, and see what happens: Download XtraDB 5.5 Source cd C:\xtradb-5.5 mkdir bld cd bld cmake .. VS08: File -> Open -> Solution -> C:\xtradb-5.5\bld\MySQL.sln Build Ended With: ========== Build: 70 succeeded, 17 failed, 2 up-to-date, 10 skipped ========== The first failure had to do with innobase: 18>Generating Code... 18>Build log was saved at "file://c:\..\innobase.dir\Debug\BuildLog.htm" 18>innobase - 9 error(s), 3 warning(s) I checked the innobase build log and found this: sql_prepare.cc ..\sql_prepare.cc(2199) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2232) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2639) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2672) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2808) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2848) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2940) : error C3861: 'gettimeofday': identifier not found ..\sql_prepare.cc(2973) : error C3861: 'gettimeofday': identifier not found ... sql_parse.cc ..\sql_parse.cc(5750) : error C3861: 'gettimeofday': identifier not found ..\sql_parse.cc(5827) : error C3861: 'gettimeofday': identifier not found The ‘gettimeofday’ function is a Unix-only (i.e., there is no direct analog of the gettimeofday() in Windows) .. more details here. So, this part of the code has simply not been ported to Windows yet. I got to looking, and it is a reported bug already: https://bugs.launchpad.net/percona-server/+bug/737895 https://bugs.launchpad.net/percona-patches/+bug/421925 https://bugs.launchpad.net/percona-patches/+bug/390156 But, do not worry. If you want to take advantage of the InnoDB enhancements found in XtraDB on Windows, you can simply use MariaDB. There are pre-built MariaDB binaries for Windows, and MariaDB contains XtraDB+ (which is XtraDB plus some additional enhancements on top of it). Hope this helps.

Speaking at MySQL Meetup in Charlotte,NC

Mon, 06 Feb 2012 14:50:30 GMT

February is going to be a busy meetup month for me. In addition to speaking in Raleigh I will visit Charlotte to speak at Meetup out there on February 23rd. Last year I visited Charlotte meetup was in the great place and we had great food and great crowd of people showing up. I will [...]

On datatypes, domains and why I think it's time we reconsidered

Mon, 06 Feb 2012 10:34:00 GMT

What's in a datatype then? A MySQL SMALLINT? A C int? An Oracle BLOB? One thing is for sure, they are not very well standardized, not even within the same environment, and much less so across them. And what does it means, really? When should I use a BLOB, when a BINARY VARCHAR and when to use a long long?A datatype defines many attributes:What data I can store in it: Only numbers? Printable alaphanumeric characters? Unicode? Binary data? An object?What I can do with a value of that particular type and how does it behave? Can I concatenate two values? Add them (that is NOT the same as concatenate! The same operator (+) may be used, but it's not the same thing!)? Save it to a file?How is it stored and represented in technical terms. And this can be argued, a C int doesn't define how it is stored on disk, but a MySQL INT does! And a BLOB is a type that is largely there just because it defines how to handle things on disk!To link this up with MySQL, let's look at the built-in MySQL datatypes. Some of them are rarely used, and for all intents and purposes, many of them are overlapping and in some cases they are there JUST BECAUSE they have different semantics and behaviour! I am, for example, not a big fan of all the different INT types that MySQL supports; tinyint, smallint mediuming, bigint all in signed and unsigned shape. Why is this useful? Yes, I know a smallint is smaller on disk so there is 2 bytes less to read from disk compared to an int. Yeah right, big deal: Look now, if that is a big deal for your application, then you have bigger problems. Which doesn't mean that using a smallint isn't a means of, in some cases, improve performance a small bit. And that this is sometimes useful. But really, this should be hidden somewhere, we just have too many datatypes. The same goes for all the different BLOB datatypes! The BLOB datatypes has another issue by the way, the names of them a a bit silly: "TINYBLOB" is a contradiction, as BLOB stands for Binary LARGE Object! A Large object that is Tiny? And a something that is at the most 255 bytes isn't really "Large" anymore, that was way back in the 1970's or so.No, I'm not a big fan of determining which datatype to use by considering storage requirements to optimize performance. That is something you might do as a performance enhancing measure after the job is done, possibly. In C, which is a language I have used longer than I care to remember, int seems to serve the role of being the default numeric integer datatype, which is not a bad thing, in particular not in C as C is a language used to write low-level stuff in, stuff that is performance sensitive, and a C int is linked to hardware specific integers.FLOAT is another issue altogether. In very few business applications is there a need for FLOAT or DOUBLE in terms of IEEE floating point values. Really! This is a scientific number format, that is also supported by the Floating point unit in most computers of today, so operations on them are pretty fast these days. But I have seen too many business applications where the developer uses a FLOAT in the database for any kind of numeric value that is not an integer, which is a distinctly bad idea! It works as long as the numbers aren't too big, and when the operations on them are reasonably simple. This is not to say a FLOAT or DOUBLE isn’t useful, just that they are more targeted for scientific programming (Hey, they are after all defined by IEEE!). What most people really should use is DECIMAL (I'm using MySQL datatypes for reference, but you know what I mean). This is a datatype that accepts (hey, big surprise coming up here) decimal numbers, fixed point decimal numbers! Operations on DECIMAL are slower though, as they aren't really supported directly by either C (which is the language that MySQL is largely written in) or by the CPU / FPU. But they are better for business applications (for scientific applications, use FLOAT or DOUBLE. And continue to write your code in Fortran by all means!)But all these datatype are increasingly used also to define the domain of the data in them! The SQL DOMAIN feature is in the SQL Standard but is largely forgotten. It is there in PostgreSQL for you to try. The advantage of this, in my mind, isn't just that you can define in more exact terms what data should go into a database column, without writing code to do that, and to make it apparent in the database schema what data is expected, it also means that we can get away from the wide array of different built-in datatypes in, say, MySQL. Instead we can stick with a few optimized ones, let them have a simple inherent domain and then we can define the application specific domains in application code. Anyone for a IP_ADDRESS datatype? A ZIPCODE datatype? An URL type. All those a reasonable requests from the application POV, but it really shouldn't (in my mind) be defined by the database server (As: What are the semantics of a ZIP-code? They are different in different countries, and the post-office can change their minds (although that is something that takes a long time I guess)). Why? As this is application specific domain! And application specifics should be in application code! Simple as that!So what datatypes are useful then, the way I look at it:RAW Data - This should be a basic type. As long as can be, or as short. Any data can be stored in it, and nearly as much as you would like.Integers - Although an integer can be considered to be a special case of a more generic number, the integer domain is so generic, and the integer is such a common and basic type that is deserves a place here.Decimal – Fixed point decimal numbers.Float - Now we are getting close to the application domain here, but these guys deserve a place here anyway, as they are so common and hooked up to the programming languages and hardware.DATETIME - Only one of them is really needed. One of DATETIME, DATE, YEAR, MONTH, TIMESTAMP, TIME etc etc. that is. Just one, very generic base type, where the specific domain is, you've heard it before, application specific!Text - Reluctantly I add this to the list. Although this is just a RAW with some limitations (usually only UTF8 or ISO8859-15 or something characters), there are some operations and attributes that are so tied up with text strings that a TEXT type is reasonable to include.And that's about it! Yes! We could add a few standard domains of top of all this, like FLOAT and DOUBLE, SMALLINT, TINYINT etc etc. But as for the basic types, this is what I would like to see. As for the weirdo TIMESTAMP semantics in MySQL, add them as triggers on your table, in application code or, best, as a specific domain (I do know that not all weirdiness of TIMESTAMP can be represented by a standard SQL DOMAIN, which is both a sign of just how weird TIMESTAMP is, and of some of the limitation of the SQL DOMAIN).I have now mentioned SQL DOMAINs a few times, so lets spend some quality time with them right now before closing up, here is a simple session using PostgreSQL 8.4:First we create a domain:CREATE DOMAIN yesno AS CHAR(1)DEFAULT 'N'NOT NULL CHECK (VALUE = 'Y' OR VALUE = 'N');Then we create a table that uses that domain:CREATE TABLE user_active(username CHAR(10), is_active YESNO);And then we check if it works using a few SQL statements:INSERT INTO user_active VALUES('foo', NULL);ERROR: domain yesno does not allow null valuesINSERT INTO user_active VALUES('foo', 'Y');INSERT 0 1INSERT INTO user_active VALUES('foo', 'N');INSERT 0 1INSERT INTO user_active VALUES('foo', 'A');ERROR: value for domain yesno violates check constraint "yesno_check"Before I wrap this up, I want to say one more thing on the issue of SQL Domains: There is an alternative n PostgreSQL, which is user defined datatypes using a CREATE TYPE SQL command. To begin with, I like domains much better as they are declarative, which means I don’t have to read a whole bunch of (usually uncommented and undocumented) code to understand what they type does. Secondly, this little warning in the PostgreSQL 9.1 manual isn’t too encouraging (page 1289): “To create a new base type, you must be a superuser. (This restriction is made because an erroneous type definition could confuse or even crash the server.)”I have more to say on types, and I will follow up with a new post on this subject eventually, but this is enough for now./Karlsson

The Agony of Big Transactions in the MySQL Binlog

Mon, 06 Feb 2012 04:31:00 GMT

Databases fail in interesting ways. About a week ago I was called in on a support case for Tungsten Enterprise. We were getting failures in which the master MySQL 5.1.50 server would run out of connections, block applications for 5 to 10 minutes and eventually trigger automated failover to a slave. Running out of database connections is a classic symptom of blocking on a shared resource inside the server. The blocked transactions hold onto their connections, which are quickly exhausted if new transactions constantly arrive from applications. So where was the hold-up?Our first clue was to notice that Tungsten Replicator was processing a huge transaction at the time of one of the failovers. To find out more I dug into the MySQL binlog using the handy mysqlbinlog utility and looked at transactions committed around the time of the failure. The following pattern quickly became apparent across hundreds of transactions. The timestamps show that the commit occurred over 8 minutes after the first statement. #120201 11:57:59 server id 313819 end_log_pos 215822862 Query thread_id=55165463 exec_time=0 error_code=0SET TIMESTAMP=1328119079/*!*/;BEGIN/*!*/;# at 215822862#120201 11:49:23 server id 313819 end_log_pos 215823092 Query thread_id=55165463 exec_time=1 error_code=0SET TIMESTAMP=1328118563/*!*/;(SQL statement)/*!*/;# at 215823092#120201 11:57:59 server id 313819 end_log_pos 215823119 Xid = 32444278496COMMIT/*!*/;The root cause turned out to be simple. The large transaction we found in the Tungsten log resulted in over 1.3Gb of data in the MySQL binlog. Mysqld took several minutes to write this into the log. Meanwhile, since transactions apply serially into the binlog, every other transaction had to wait instead of just committing and releasing its connection. New transactions quickly consumed the spare connections up to the max_connections limit. Problem explained. The MySQL binlog is a wonderful thing but the fact that it is a serial resource has important consequences for busy servers that use replication. First, if you run out of connections, look at the binlog. Tungsten makes large transactions fairly easy to detect because they are fragmented in the replicator log. We noticed a transaction with 1300 fragments, which tipped us off to the problem. However, you can also find the problem by looking at the binlog directly as I did above. Or you can use any of several tools to analyze the binlog and look for periods of time with no transactions committed. Second, big transactions are a problem for MySQL. Admin operations or application bulk loads through web APIs can easily create huge numbers of updates. Break them up into pieces and do a chunk at a time. This is also better for slaves, which may end up lagging badly when large transactions reach them. Increasing max_connections to a high number is incidentally not a solution. It just makes problems with transactions less visible and will not help anyway if you have high transaction throughput. Third, if you have updates or deletes that affect many rows, use statement replication. Row replication is a great thing. We use it regularly in Tungsten deployments, especially for heterogeneous replication, e.g., from MySQL to Oracle. However, we have also seen customers write 5Gb or more into the binlog without even realizing it. Statement replication can write as little as a single statement. This flexibility is one of the outstanding features of MySQL replication.A possible fourth suggestion is to put the binlog on faster storage. However, it is unclear whether that would help--the binlogs in this case were already on fast, separate disk storage with large caches and sync_binlog=0. It seems MySQL writes large transactions rather slowly into the binlog for reasons that may not have much to do with storage speed. I hope to understand why that is at some later time. Perhaps somebody reading this article can suggest a reason.

Managing MySQL with Percona Toolkit by Fr�d�ric Descamps

Sun, 05 Feb 2012 15:58:15 GMT

Frédéric Descamps of Percona. Percona Toolkit is Maatkit & Aspersa combined. Opensource and the tools are very useful for a DBA. You need Perl, DBI, DBD::mysql, Term::ReadKey. Most tools are written in Perl, and whatever is in Bash is being re-written in Perl. There is also a tarball or RPM or DEB packages. Know your environment. The hardware & OS are crucial for you to know. How much memory/CPU do you use? Do you use swap? Is this a physical/virtual machine? Do you have free space? What kind of RAID controller? Volumes? Disk? What about the network interfaces? What IO schedulers are used? Which filesystem is the data stored on? To answer all that, just use pt-summary. Know your MySQL environment. Version? Build? How many databases? Where is the data directory? What about replication? What are key InnoDB settings? Storage engine in use? Index type? Foreign keys? Full text indexes? To answer all this and more use pt-mysql-summary. pt-slave-find shows you the topology and replication hierarchy of your MySQL replication instances. An inventory of replicas! Where is my disk I/O going? Use pt-diskstats which is an improved iostat. There is pt-ioprofile but it can be dangerous in production. Now its time to get more intimate with your database. Let’s try to find the answer to these questions: how are the indexes used? Are there duplicate keys? Which queries are eating most of the resources? You can use pt-duplicate-key-checker to check for duplicate/redundant indexes or foreign keys. pt-index-usage can tell you which indexes are unused. If you think you have bad SQL, check out pt-query-advisor. You can use pt-query-digest to analyze the slow query log and show a profile of the workload. You mostly use this with slow query logs & tcpdump’s. Be careful when you have dropped packets — results may tend to be fake then! After all this, its time to maintain your environment. pt-deadlock-logger checks InnoDB status to log MySQL deadlock information. It needs to run continually to capture things. pt-fk-error-logger extracts and logs MySQL foreign key errors. pt-online-schema-change to alter tables. It makes a “shadow copy” and swaps them. Extremely useful for large, long-running ALTER. Facebook uses the same technique. Validate your upgrades as upgrades are the leading cause of downtime. Are queries using different indexes? Is query execution plan different? New errors? See pt-upgrade for this. Best to run this on a third machine (i.e. the old machine and a new machine to see how it goes). Verify replication integrity – pt-table-checksum. Perform an online replication consistency check or checksum MySQL tables efficiently on one or many servers. Use it routinely (mandatory for 95% of MySQL users). Put it in a weekly crontab. Repair differences with pt-table-sync. Repair out-of-sync replicas – pt-table-sync Measure delay acfurately – pt-heartbeat Deliberately delay replication – pt-slave-delay Watch & restart MySQL replication after errors – pt-slave-restart When there are problems, get the symptoms when it hurts. Look at pt-stalk (wait for a condition to occur them begin collecting data – eg. everytime the threads go over 2,000 you have a problem, so it collects stuff – it calls pt-collect), pt-collect (collect information from a server for some period of time), and pt-sift. pt-mext looks at many samples of MySQL SHOW GLOBAL STATUS side-by-side. Default STATUS shows counter since the MySQL instances started. It is very helpful to see a delta of recent activity. The future: pt-query-digest will do query reviews; pt-stalk will do “magical fault detection algorithm”. Its all opensource and its all on Launchpad at lp:percona-toolkit. Related posts:Practical MySQL Indexing guidelines by Stéphane Combaudon Replication features of 2011 by Sergey Petrunia MySQL synchronous replication in practice with Galera by Oli Sennhauser

Replication features of 2011 by Sergey Petrunia

Sun, 05 Feb 2012 14:32:45 GMT

Sergey Petrunia of the MariaDB project & Monty Program. MySQL 5.5 GA at the end of 2010. MariaDB 5.3 RC towards the end of 2011 (beta in June 2011). MySQL 5.5 is merged to Percona Server 5.5 which included semi-sync replication, slave fsync options, atuomatic relay log recovery, RBR slave type conversions (question if this is useful or not), individual log flushing (very useful, but not many using), replication heartbeat, SHOW RELAYLOG EVENTS. About 2/3rds of the audience use MySQL 5.5 in production, with only 2 people using semi-sync replication. MariaDB 5.3 brings replication features brings group commit in the binary log, which is merged into Percona Server 5.5. Checksums for binlog events which is merged from MySQL 5.6. Sergey goes in-depth about the group commit for the binary log. To find out a little more about MariaDB replication changes, see Replication in the Knowledgebase. There are several implementations of group commit. Facebook started it, followed by MariaDB & Oracle. Percona 5.5 is GA so the feature is there, its not in MySQL 5.6 (yet?), and MariaDB 5.3 is where its at. Seems like the MariaDB implementation is the best so far – refer to the Facebook benchmark performed by Mark Callaghan. Annotated RBR poses a compatibility problem. MariaDB 5.3 has annotate_rows, while MySQL 5.6 has rows_query event. They are different events. So you cannot have a MariaDB 5.3 master and a MySQL 5.6 slave at this moment. So MySQL 5.6 will have a flag to mark “ignorable” binlog events which will be merged into MariaDB and this will make binary logs compatible again. There is now also optimized RBR for tables with no primary key. MySQL 5.6 also has crash-safe slave (replication information stored in tables). Crash-safe master (binary log recovery if the server starts & sees the binary log is corrupted). Parallel event execution is something that is new in MySQL 5.6 which is the most important feature for Sergey. Pre-heating: There is mk-slave-prefetch (famous quote: “Please don’t use mk-slave-prefetch on #MySQL unless you are Facebook.”). There is replication booster by Yoshinori Matsunobu. There is a Python version of mk-slave-prefetch that Facebook uses. Related posts:MariaDB 5.3 query optimizer by Sergey Petrunia Where is MariaDB today? Building simple & complex replication clusters with Tungsten Replicator by Giuseppe Maxia

MySQL Creatively in a Sandbox by Giuseppe Maxia

Sun, 05 Feb 2012 13:57:47 GMT

Giuseppe Maxia of Continuent and long time creator of MySQL Sandbox. Only works on Unix-like servers. Works with MySQL, Percona & MariaDB servers. MySQL server has the data directory, the port and the socket – you can’t share these. To use it: make_sandbox foo.tar.gz. Then just do ./use. $SANDBOX_HOME is ~/sandboxes. You can also create ~/opt/mysql/ and if you have MySQL 5.0.91 binary in that directory, you can just do “sb 5.1.91″. Sandbox has features to start replication systems as well. You can have varying master/slave setups with varying versions as well (good idea to test from MySQL -> MariaDB master->slave for migration). You can now also play with tungsten-sandbox, which is a great way to start playing with Tungsten Replicator (see documentation and tungsten-toolbox). There is apparently also a MySQL Cluster sandbox tool that someone is working on. Related posts:Building simple & complex replication clusters with Tungsten Replicator by Giuseppe Maxia MySQL HA reloaded by Ivan Zoratti MySQL synchronous replication in practice with Galera by Oli Sennhauser

Optimizing your InnoDB buffer pool usage by Steve Hardy

Sun, 05 Feb 2012 13:29:02 GMT

Steve Hardy of Zarafa. Work that has been done to make Zarafa better. Why do you optimise your buffer pool? To decrease your I/O load. How can you do it? Buy more RAM, page compression, less (smaller) data, rearrange data. MariaDB or Percona Server allows you to inspect your buffer pool (unsure if this is now available in MySQL 5.6). Giuseppe in the audience says this is available in MySQL 5.6, but Steve used this on MariaDB 5.2. Strategies to fix it: Make records smaller. Remove indexes if you can use others almost as efficiently. Make records that are accessed around the same time have a higher chance of being on the same page. Use page compression. Buy more RAM. Try Batched Key Access (BKA) in MariaDB 5.3+. Best to view the presentation since there are specific examples that speak about how Zarafa solves their problems like a user trying to sort their email, etc. Related posts:Batched Key Access Join Previews available MariaDB 5.3 query optimizer by Sergey Petrunia MariaDB/MySQL users in Paris & Brussels

Practical MySQL Indexing guidelines by St�phane Combaudon

Sun, 05 Feb 2012 13:01:06 GMT

Stéphane Combaudon of Dailymotion. Index: separate data structure to speed up SELECTs. Think of index in a book. In MySQL, key=index. Consider that indexes are trees. InnoDB’s clustered index – data is stored with the Primary Key (PK) so PK lookups are fast. Secondary keys hold the PK values. Designing InnoDB PK’s with care is critical for performance. An index can filter and/or sort values. An index can contain all the fields needed for the query you don’t need to go to the table (a covering index). MySQL only uses 1 index per table per query (not 100% true – OR clauses), so think of a composite index when you can. Can’t index TEXT fields (use a prefix). Same for BLOBs and long VARCHARs. Indexes: speed up queries, increases the size of your dataset, slows down writes. How big is the write slowdown? Simple test by Stephane, for in-memory workloads he says adding 2 keys makes performance 2x worse; for on-disk workloads he says its 40x worse. Never neglect the slowdown of your writes when you have an index. There is a graph in the slidedeck. What is a bad index? Unused indexes. Redundant indexes. Duplicate indexes. Indexing is not an exact science, but guessing is probably not the best way to design indexes. Always check your assumptions – EXPLAIN does not tell you everything, time your queries with different index combinations, SHOW PROFILES is often valuable. Slow query log is a good place to start. Many slides with examples, so I hope Stephane posts the deck soon. If possible, try to sort & filter (an index is not always the best for sorting). InnoDB’s clustered index is always covering. SELECT by PK is the fastest access with InnoDB. An index can give you 3 benefits: filtering, sorting, covering. See Userstats v2 - you need Percona Server or MariaDB 5.2+. See also pt-duplicate-key-checker to find redundant indexes easily. See also pt-index-usage to help answer questions not covered by userstats. Related posts:Sphinx user stories by Stéphane Varoqui Managing MySQL with Percona Toolkit by Frédéric Descamps MySQL Full Text Search by Alex Rubin

MySQL synchronous replication in practice with Galera by Oli Sennhauser

Sun, 05 Feb 2012 12:30:42 GMT

Oli Sennhauser of FromDual. Synchronous multi-master replication with the Galera plugin. Your application connects to the load balancer and it redirects read/write traffic to the various MySQL Galera nodes. Tested a setup with 17 SQL nodes and you can have even more. Scaling reads and also a little bit for scaling writes is what Galera is good for. If one node fails, the other two nodes still communicates with each other and the load balancer is aware of the failed node. Why Galera? There is master-slave replication but its not multi-master, and its asynchronous and you can get inconsistencies. There is master-master replication but its asynchronous and can have inconsistencies and conflicts if you write on both nodes. MHA/MMM/Tungsten are not providing new technology but are based on the MySQL replication technology. MySQL Cluster is another solution but its not InnoDB storage & your need new know-how for Cluster. Also Cluster has problems with fast JOINs. Active/Passive failover clustering, but too often you have resources idling. Schooner being closed & expensive is hard to know much about what they’re doing. Galera is synchronous & based on InnoDB (others should in theory be possible). Active-active real multi-master topology. True parallel replication on row level. Cluster speaks with each other. There is no slave lag. Won’t lose transactions. Read/write scalability, write throughput can be improved but can’t scale in the way like MySQL Cluster. Disadvantages? Its not native MySQL binaries/sources but a patch. Codership provides binaries. Higher probability of deadlocks. When you do a full sync (like when a node comes back after downtime), one node is blocked. This is why the minimum you need a 3-node cluster. Also if you do a full sync with a database larger than 50GB, the recommended method is to use mysqldump (which can be very slow). You can use rsync. Percona is working on xtrabackup to do a full sync between nodes. Setup: 3 nodes are recommended. Or just 2 nodes and one for garbd (Galera Arbitrator Daemon). 2 nodes works but pay attention to a split brain scenario. Go to the Codership website, download their binaries and wsrep (the Galera plug-in). Create your own user on all nodes (don’t use the default root user). You then need to configure my.cnf (there have been discussions for a galera.conf, but Oli just uses my.cnf). Galera works only with InnoDB, so in my.cnf make the default storage engine InnoDB (don’t for example, by accident have MyISAM tables). The demo has a strange Galera start script, but its not been easy to work. Just start MySQL usually like you would do. SST is Snapshot State Transfer (SST). Its the initial full sync between the 1st and the other node. SST blocks the donor node (hence why you need 3 nodes). With Galera v2.0, there is also incremental state transfer. It should be GA in February 2012. You can get deltas as opposed to the full sync. You can configure which will be the donor node. Currently there are 27 variables about Galera in v1.1. You can do it just by doing SHOW GLOBAL VARIABLES LIKE ‘wsrep%’;. The plugin itself, wsrep_provider_options has plenty of options & plenty of room for tuning. SHOW GLOBAL STATUS LIKE ‘wsrep%’; currently has 38 status information fields in Galera v1.1. For load balancing, you can do it in your application (on your own). You can also use Connector/J which provides load balancing. There is also a PHP MySQLnd that works. Related posts:MySQL HA reloaded by Ivan Zoratti Ticketmaster thrives on MySQL Replication New MySQL 5.6 Features by Oli Sennhauser

Optimising SQL applications by using client side tools by Mark Riddoch

Sun, 05 Feb 2012 11:54:14 GMT

Mark Riddoch of SkySQL. This was a talk about the future in general. What people would like. Etc. Not about something that exists yet, hence the sparse notes. Trace statements are good for the “why”. You move on to the debugger, but what is a useful SQL debugger? Profilers addresses the “when”. SQL developer tools: manually run queries (traditional route for development, effective way to test SQL statements, some indication of performance), server logs (alerts developers of serious issues like the slow query log – identify poorly written queries, requires server access, not effective in a cloud environment (?)), external monitoring (network sniffing of connection packets, Ethereal dissectors – allow individual connections to be traced, no server access, privileged network access, complex to interpret, possibility of packet loss), intrusive tools (insert “proxy” between client & server to intercept all traffic, imposes delay & requires setup modification). Client-side tools – client modification or hook via connectors (query logging, profiling). The Java connector has profiling. Should there be a connector slow query log? So there’s no requirement for server access and its per client rather than per server. Plans: roll out connector query logs for Java, C, and scripting languages built on Connector/C. SQL Parser for report generation/fuzzy matching. Create post-processing tools. Query log comparisons – detect fluctuation in execution time, plan flip alerting, replay log. Related posts:Managing MySQL with Percona Toolkit by Frédéric Descamps Open source tools to run a small-medium sized business Practical MySQL Indexing guidelines by Stéphane Combaudon

MySQL Cluster by Ralf Gebhardt

Sun, 05 Feb 2012 11:25:32 GMT

Ralf Gebhardt of SkySQL. Cluster: shared nothing architecture (no single point of failure), synchronous replication between nodes, ACID transactions, row level locking. In-memory storage (some data can be stored on disk, but indexes must be in-memory). Checkpointing to disk for durability. It supports two types of indexes – ordered T-trees, unique hash indexes. Online operations like adding node groups, software upgrades, table alterations. Quick standard architecture diagram displayed about MySQL Cluster. Network partitioning protocol is designed to avoid a split brain scenario. Is there at least one node from each node group? If not then this part cannot continue – graceful shutdown. Are all nodes present from any node group? If so, then this is the only viable cluster – continue to operate. Ask the arbitrator – the arbitrator which parts will continue if no arbitrator is available the cluster shuts down. Durability – in order for a node to recover fast some data is stored locally. The REDO log is synchronized by global checkpoints (GCP). The DataMemory is synchronized by local checkpoints (LCP). I agree with Ralf — almost impossible to talk about NDB in 25 minutes. Its very deep, you’d need at least three hours to grasp it well. Related posts:MySQL synchronous replication in practice with Galera by Oli Sennhauser Practical MySQL Indexing guidelines by Stéphane Combaudon Managing MySQL with Percona Toolkit by Frédéric Descamps

Building simple & complex replication clusters with Tungsten Replicator by Giuseppe Maxia

Sun, 05 Feb 2012 11:01:26 GMT

Giuseppe Maxia of Continuent. MySQL replication is single threaded. Multi-master replication is complex with MySQL. Circular replication works but is very fragile. Once you’ve achieved the feat, how do you avoid conflicts? The lack of global transaction ID today also means you may have slaves that may not be synchronized fully. Finally, some people like to replicate to PostgreSQL, Oracle and MongoDB. This is where Tungsten Replicator comes into place. Opensource. 100% GPLv2. You can do easy failover (no need to synchronize the slaves manually when a master dies), have multiple masters, multiple sources to a single slave, conflict prevention, parallel replication, and replicate to Oracle/PostgreSQL database (heterogeneous replication – Oracle->MySQL is not opensource, but MySQL->Oracle is). Parallel replication: ability to replicate with multiple thread at once. Sharded by database (all big sites already use this naturally). Good choice for slave lag problems. Bad choice for single database projects. In their tests to measure slave catch up time between standard MySQL & Tungsten slave with 1hr of sysbench and some 130GB of data, MySQL replication takes 4.5h to catchup, and Tungsten takes less than 1 hour to catchup. This comparison was not made with MySQL 5.6 (the same parallel replication feature at this moment in 5.6 is quite buggy). No need to install Tungsten on the master (Tungsten needs to be on slave). Replication can revert to native salve with 2 commands, but the failover is an issue — so install Tungsten on both! Conflict prevention. When you have multiple masters you are tempted to do many things that you shouldn’t. Tungsten provides the multi-master topology. Tungsten can help you avoid conflicts. Decide you want to shard your information by database – Tungsten can enforce such rules. Define the rules, applied to either the master or the slave and you can then state to make replication fail or drop silently or drop with a warning. You can create clusters very quickly with Tungsten. You can install from a centralized point to many servers. You need Java, Ruby, SSH access and a MySQL user with all privileges (used for replication). Related posts:MySQL Creatively in a Sandbox by Giuseppe Maxia MySQL HA reloaded by Ivan Zoratti Replication features of 2011 by Sergey Petrunia

How to offload MySQL server with Sphinx by Vladimir Fedorkov

Sun, 05 Feb 2012 10:29:05 GMT

Vladimir Fedorkov of Sphinx. Presentation started out with a very nice presentation of candies to all the audience members. What is Sphinx? Another (C++) daemon on your boxes. Can be queried via API (PHP, Python, etc.) or MySQL-compatible protocol and SQL queries (SphinxQL). Some query examples are in the slides, here’s one about SphinxSE in the KB. MyISAM FTS is good but becomes slow with half a million documents. InnoDB has FTS now but he’s not tried it (and neither has anyone in the audience to see it compare with MyISAM FTS). Geographical distance is the distance measuring the surface of the earth (two pairs of float values – latitude, longitude). In Sphinx, there is support for GEODIST(Lat,Long,Lat2,Long2) in Sphinx. Segments are good for price ranges on a site, date ranges, etc. Use INTERVAL(field, x0, x1, …, xN). Keep huge text collections out of the database. sql_field = path_to_file_text. Tell Sphinx to index text not from MySQL but out in the filesystem. Keep the metadata inside the database but keep the actual data outside of the database. max_file_field_buffer needs to be set properly. You can do proximity search with Sphinx — find the words “hello world” within a ten word block, for example. Resources: the documentation, a book by O’Reilly: Introduction to Search with Sphinx: From installation to relevance tuning (sold out at the FOSDEM O’Reilly booth!), and their community page including wiki, forum, etc. Related posts:MySQL Full Text Search by Alex Rubin Sphinx user stories by Stéphane Varoqui Morning sessions at MySQL MiniConf

MariaDB 5.3 query optimizer by Sergey Petrunia

Sun, 05 Feb 2012 09:59:23 GMT

Sergey Petrunia of the MariaDB project. What exactly is not working in MySQL? MySQL is poor at decision support/analytics. With large datasets you need special disk access strategies. Complex queries like insufficient subquery support and big joins are common int he MySQL world. DBT-3 is used, scale=30, with a 75GB database and run a query “average price of item between a range of dates”. Query time took some 45 minutes to execute. Why? Run iostat -x to see what is going on. See that the CPU is mostly idle, so its an IO-bound load. Next you run SHOW ENGINE INNODB STATUS and you’ll see how many reads per second is happening. Possible solution is to get more RAM or get an SSD (good to speedup OLTP workloads, but analytics over data is probably not viable since SSDs are small and not cheap). The MySQL/MariaDB solution to the above problem is improved disk access strategies: multi-range read (MRR) and batched key access (BKA). In MariaDB, MRR/BKA need to be enabled (they are not turned on by default). The query time only took 3 minutes 48 seconds, which is some 11.8x faster than the previous 45 minutes.If you look at EXPLAIN output, its almost as same as before, expect the Extra filed. iostat -x will now show some CPU load, svctm down as well (so its not random disk seeks anymore — some 8ms seek time on a regular 7,200rpm disk), SHOW ENGINE INNODB STATUS will show some 10,000 reads per second rather than the previous 200. If you are on Fedora, check out the Systemtap feature to look at I/O patterns. stap deviceseeks.stp -c “sleep 60″. Subqueries handling in MariaDB 5.3: check out the Subquery Optimizations Map. Only about 10% of the audience use optimizer hints in MySQL. Related posts:MariaDB 5.1.42 released! MariaDB 5.1.44 released Where is MariaDB today?

MySQL HA reloaded by Ivan Zoratti

Sun, 05 Feb 2012 09:32:10 GMT

MySQL HA reloaded – old tricks and cool new tools to guarantee high availability to your MySQL Servers by Ivan Zoratti of SkySQL. This talk is a little longer, so check out: HA Reloaded – many ways to provide High Availability. The slides are already online. Questions to ask: which level of high availability do I need? Do I require no loss of data? Do I need failover or is switchover enough? Can I provide a reasonable service when a component is down? Remember, five nine’s high availability also means a lot of infrastructure costs. Other things to clarify: availability vs scalability. HA costs. HA for your entire architecture, not just for your database servers. Review your SLAs. The best high availability solution today is combined solutions. MySQL replication – asynchronous & semi-synchronous (lots of people use MySQL 5.5, about 4 people in room were on Percona Server – question asked due to semi-sync replication only being available in 5.5 & greater), there are pros & cons of row based replication vs statement based replication. MySQL Replication via Multi-Master replication Manager (MMM). Features such as monitoring, automatic failover, data backup & resync. Unfortunately, it has some problems with the stability & automatic failover. The project is not improved anymore, so there are other solutions that you should consider today. MySQL Replication with MHA is a preferred solution. Something to consider: –read-only=1 and log-bin on slaves. Master IP failover. FIltering rules. Multi-tier replication. Tungsten Replicator – open source, heterogenous replication. Truly multi-master and fan-in with Global ID. Per-schema multi-thread. You can also use it to replicate to Postgresql, Oracle and other databases. There is also Tungsten Enterprise. Synchronous replication with DRBD is typical for active/standby environments. People don’t really like this because they feel that there is a server doing nothing. You can always do it active/passive. It works with InnoDB only. Synchronous replication with Galera works for InnoDB. Its multi-master with no SPOF. Its new/young technology so you may find some issues with it. Application failover must be managed, but the conflict resolution is quite tricky (when you commit a transaction you might be fine, but you may have transactions that are removed due to conflicts). There is a commercial SchoonerSQL that provides synchronous master-slave replication for InnoDB. Its defined explicitly as a master-slave solution. Active/Passive clusters using Shared Storage. Points to consider are the fact that redundancy & replication must be guaranteed by the shared storage. InnoDB only. What about filesystems? Virtualized environments – data storage, high availability & load balancing are provided and managed by the virtualized software. The faults are handled by the software, not the database. There is also geographical replication for disaster recovery, having a master-master asynchronous replication is used to update the backup data centre. There is also storage snapshots for disaster recovery (not-specific to MySQL, its storage systems based, use only InnoDB). There is also MySQL Cluster but there is another presentation about this later at FOSDEM. Very nice closing slide, “The absolutely necessary comparison chart” which some may disagree, but Ivan thinks is the best way forward. Related posts:MySQL at Google Ticketmaster thrives on MySQL Replication New MySQL 5.6 Features by Oli Sennhauser

Sphinx user stories by St�phane Varoqui

Sun, 05 Feb 2012 09:01:25 GMT

Stephane Varoqui, Field Services SkySQL, Vlad Fedorkov, Director of PS, Sphinx Inc, Christophe Gesche, LAMP Expert, Delcampe, Herve Seignole, Web Architect, Groupe Pierre & Vacances Center Parcs – this is a big talk! Pros: Filtering takes place on attributes in separate tables. Rely on the optimizer choice. HASH JOIN can help (MariaDB 5.3). Table elimination can help (MariaDB 5.2). ICP Index Condition Pushdown can help (MariaDB 5.3/MySQL 5.6). Max 80M documents at Pixmania, all queries come in less than 1s using 128GB of RAM (MariaDB 5.2). At PAP.fr, there is 16GB RAM with MariaDB 5.2. Cons: CPU intensive (replication with many slaves). Need covering indexes to cover various !filter !order. Join & sorting cost on lazy filtering. The more indexes you have in the system, the more you need to increase the main memory of the server. Keep the Btree’s in memory. What about denormalized schemas? Not really CPU intensive, just IO. Can go to disk, full partition scan with filtering taking place on record order using covering index. Can shard but not that easy. Use the spider storage engine or shard-query. Can use memory engine for isolation. There are cons like duplicate data, duplicate indexes, missing material views, merge index cost, impact on write performance, and can consume a lot of memory with many indexes. MySQL can push hardware, so read less/do less/read serialized/map reduce to get better latency. Chose data type wisely, replace string with numeric, vertical & horizontal sharing, snowflake compression (combination of attributes, build a table of the combination and replace it with an ID). If you are lazy, just use Sphinx! Sphinx is just another daemon that can serve queries. Its easy to setup, easy to scale, storage engine makes it accessible to current MySQL users, API in core MariaDB (SphinxSE), SphinxQL, SphinxSE is transparent to the application layer of MySQL protocol. Demo done using the Employees DB. Pierre & Vacances – Centerparcs. Free text search, they use MariaDB using Levenshtein UDF implementation. Went live 01/2011. First implementation of Sphinx (12 indexes). Its grown, they use PHP API. The new goal is to never send an empty result. 1 index per website/market, with a total of 15 million docs. Index built on standalone server. Using internal gearman job schedule to generate index before cache generated. Current monitoring is via Nagios & perl, but the next step is to use Monyog & MariaDB INFORMATION_SCHEMA plugin. Delcampe is an auction website with 45M ‘active’ items. Its dedicated for collectors. 3 string fields, and 15 attributes. 40-120K new items daily. Started with mysql fulltext in 2007, moved to Sphinx in 2008. There was a need to have more filters. Now they have 5 sphinx servers + 1 MySQL server. HAproxy to load balance. Related posts:How to offload MySQL server with Sphinx by Vladimir Fedorkov MySQL Full Text Search by Alex Rubin Abusing MySQL (& thoughts on NoSQL)

New MySQL 5.6 Features by Oli Sennhauser

Sun, 05 Feb 2012 08:33:08 GMT

First talk at FOSDEM MySQL Devroom by Oli Sennhauser, of FromDual. Quick notes/liveblog of the talk, plus links from a quick search. New Release Model: starts with at least in beta quality, milestone releases are RC quality (every 3-6 months), between milestones new features are allowed, GA releases every 12-18 months, no more than 2 releases in active support. There is also MySQL Labs, in where features can make it into a release model, but not necessarily. Oli’s guess: MySQL 5.6 GA in April 2012 (Collaborate) or June 2012. As a consequence, MySQL 5.0 and MySQL 5.1 will be EOL probably by April this year. New improvements in partitioning: explicit partition selection, exchanging partitions (good for ETL jobs). New improvements in InnoDB: InnoDB INFORMATION_SCHEMA has got some new entries. Buffer (INNODB_BUFFER), Fulltext (INNODB_FT), Metrics (INNODB_METRICS), Data Dictionary (INNODB_SYS). The InnoDB Monitor is now obsolete. innodb_purge_threads can be set > 1. Kernel mutex split which should lead to improved concurrency. Persistent optimizer statistics (since InnoDB used to do random dives, so when you do a mysqldump, things can be different) – you can do SET GLOBAL innodb_analyze_is_persistent = 1; (read more: InnoDB Persistent Statistics at last, InnoDB Persistent Statistics Save the Day). New improvements in optimizer: ORDER by on non-indexed columns, Multi Range Read (MRR), Index Condition Pushdown (ICP), query execution plan for DML statements, Batched Key Access (BKA). There is also optimizer tracing. There is now some new instrumentations in Performance Schema (introduced in MySQL 5.5). There is also improved replication features, but that will be covered in a later talk. Fractional seconds (microseconds) is introduced in MySQL 5.6. GET DIAGNOSTICS for stored procedures. Pluggable authentication (socket). memcached/InnoDB still in labs, hasn’t made it to a milestone release. Some 400+ bugs have been fixed. Lots of cleanup’s (old unused variables, commands removed). Pay attention to MySQL 5.6 incompatible changes when you plan to upgrade. Probably some good resources: What’s New in MySQL 5.6, MySQL 5.6 (Early Access Features) – InnoDB & Replication. Related posts:MySQL HA reloaded by Ivan Zoratti MariaDB/MySQL users in Paris & Brussels Some MySQL-related links

Consistent transactions between storage engines

Sun, 05 Feb 2012 01:40:12 GMT

You may not realize it, but in MariaDB 5.2 and earlier and in MySQL up to version 5.5, START TRANSACTION WITH CONSISTENT SNAPSHOT does not give any guarantees of consistency between different storage engines. For example, suppose you have two transactions which run in parallel: Transaction T1: BEGIN; SET @t = NOW(); UPDATE xtradb_table SET a= @t WHERE id = 5; UPDATE pbxt_table SET b= @t WHERE id = 5; COMMIT; Transaction T2: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ; START TRANSACTION WITH CONSISTENT SNAPSHOT; SELECT t1.a, t2.b FROM xtradb_table t1 INNER JOIN pbxt_table t2 ON t1.id=t2.id WHERE t1.id = 5; In the above case, it is possible, even with a "consistent" snapshot, to see the changes in a transaction only in InnoDB/XtraDB tables, and not in PBXT tables. Naturally, it would be much better if the changes were visible no matter the storage engine used by the table. Thankfully this is possible, thanks to the introduction in MariaDB 5.3 of group commit and an enhanced storage engine API for COMMIT which allows storage engines to coordinate commit ordering and visibility with each other and with the binary log. With these improvements, the START TRANSACTION WITH CONSISTENT SNAPSHOT statement in MariaDB 5.3 has been enhanced to ensure consistency in-between storage engines which support the new API. Currently the storage engines which support this are XtraDB (MariaDB's enhanced version of InnoDB) and PBXT. In addition, the binary log, while not a storage engine as such, also supports the new API and can provide a binlog position consistent with storage engine transaction snapshots. This means that with transaction isolation level at least REPEATABLE READ, the START TRANSACTION WITH CONSISTENT SNAPSHOT statement can be used to ensure that queries will see a transaction-consistent view of the database between storage engines. No longer is it possible for a query to see the changes from some transaction T in XtraDB tables without also seeing the changes T makes to PBXT tables. More information on this can be found in the AskMonty Knowledgebase .

OurSQL Episode 77: Removing Evil Queries, part 1

Sat, 04 Feb 2012 15:42:15 GMT

Registration for Percona Live: MySQL Conference and Expo is open! The conference is from Tuesday, April 10th through Thursday, April 12th. Early bird pricing ends March 12th, 2012. Use code PL-pod and save 10% off the early bird prices!. The 1st Latin American Conference about MySQL, NoSQL and Cloud technologies will be held in Buenos Aires in June. It is called the MariaDB NoSQL & Cloud Latin American Conference and we'll bring you more information as it becomes available. read more

Stripping Comments so Query Cache Works in MariaDB and XtraDB

Fri, 03 Feb 2012 22:16:54 GMT

I recently noticed both MariaDB and XtraDB (not MySQL yet) have a (newer) variable query_cache_strip_comments. This variable is great for those who want to append comments to various queries, but still want the query cache to be able to serve such queries. Unfortunately, with MySQL, this is not currently possible. In the past, I wrote a post on using MySQL Proxy which described a technique of monitoring queries through the proxy by appending IP addresses to the queries so one could track where they originated from. However, one pitfall to that was the MySQL query cache *does not* ignore the comment and treats them all as different queries (see the user comments for further discussion). (I did subsequently enhance that functionality implementing the SHOW PROXY PROCESSLIST command (often used in the Proxy Admin module), in large part because of this limitation.) To enable it (in MariaDB 5.3+ and XtraDB), just add query_cache_strip_comments under the [mysqld] section in your my.cnf file and restart mysqld. Alternatively, you can also set it dynamically: mysql> set @@global.query_cache_strip_comments=1; Query OK, 0 rows affected (0.04 sec) mysql> show global variables like 'query_cache_strip_comments'; +----------------------------+-------+ | Variable_name | Value | +----------------------------+-------+ | query_cache_strip_comments | ON | +----------------------------+-------+ mysql> select version(); +---------------+ | version() | +---------------+ | 5.3.3-MariaDB | +---------------+ Fwiw, from examining the source code, both implementations seemed to differ, but the end result is the same, and it’s a welcome addition, if you ask me.

Forums.MySQL.Com as a gym for mental exercise

Fri, 03 Feb 2012 21:21:40 GMT

The MySQL Forums seem to be have been around forever. There are groups for announcements, performance, InnoDB, Cluster, Replication, Certification, Events, questions in German, partitioning, and just under a hundred MySQL related subjects in total. Most folks in the MySQLverse seem to regularly lurk in one or more of the forums. There area great source of information and sometimes humor. But are you using the forums to help build your MySQL mental muscle? Ask yourself how would you solve the problem or how would you guide a junior DBA or developer to fix an issue posted in a forum. Try reading the groups that are not in your regular reading list. Is there a question in there you can not answer even after a peek at the documentation? After running MySQL for a few years, it is easy to skip over the problems of novices found in the Newbie section. But like the old story of the Emperor’s New Clothes, a new set of eyes can point out things we do not notice. Recently someone in the Newbie group wanted to set up a round robin for inserts over eight disks in an attempt to get 8x performance gain. Hmmm, if that did scale would it scale linearly(1) or close to it? Too bad I do not have the hardware or spare cycles to give it a try. Or question on how to remove a ‘width=xxx’ pattern from a text field. Hmm, the old GIGO rules would dictate that some sort of ETL tool or script remove the unwanted chaff before it goes into the database. Do you use regular expressions in a PHP, Perl, or Java hack? Could a column oriented editor let you WYSIWYG the data? Maybe a trigger on insert that culls the unwanted width data? Hey, does your data have anything hidden in a text filed that should not be there and how best to get rid of it? So read through a few forums and use them to exercise you mind. And if you see something in a question that you can answer, please do so. Nothing scales linearly that you need to scale linearly. At least not on your budget and or schedule.

Liveblogging: Performance is Overrated, by Mark Callaghan

Fri, 03 Feb 2012 18:42:48 GMT

Mark Callaghan speaks at the New England Database Summit about how data manageability is more important than performance. Peak performance is thrown out, 95%-98% is important. Variance shouldn't be large. Data manageability is rate of interrupts per server for the operations team. Rate of server growth much bigger than rate of new hires for the systems teams. A lot of the db team is from University of Wisconsin-Madison! Why MySQL? Because it was there when he came. Mark and ops/engineering peers made it scale 10x. He likes MySQL for OLTP, InnoDB is "An amazing piece of software." They can get 500,000 qps using a cached workload, which is on par with memcached. What Facebook really does is OLTP for the social graph. The workload is secondary indexes, index-only queries, small joins but most queries use one table, multi-row transactions, majority of workload does not need SQL/optimizer, they do a physical and logical backup. Most of this does not require SQL [blogger's note - they built Cassandra]. Why is the grass greener on the other side? automated replacement of failed nodes, less downtime on schema changes and/or fewer schema changes, multi-master, better compressions, etc. Circa 2010, 13 million queries per second, 4 ms reads, 5 ms writes, 38GB peak network per second, etc. Why so many servers? Big data high queries per seconds. They add servers to add IOPS, so they're interested in compression and flash, so they can get more IOPS. If they do remain on disk, write-optimized dbs are interesting too. About 10 people on the db team, which is very small for a company that size. How to scale MySQL? Fix stalls to make use of capacity, improve efficiency to use fewer queries/fewer data. Fixing stalls doesn't make MySQL faster, makes it less slow. [blogger's note - I stopped taking notes here because this is a rehash of the "How Facebook Does MySQL" talk that has been done over and over...] [restarted when he started talking about data manageability again] How Facebook got it's data manageable. pylander- sheds load during a query pileup - kills dup queries, limits # of queries from some specific accounts -- take off on Highlander: there can be only one. dogpile - collects data during a query pileup - gets perf counters and list of running queries, generates HTML page with interesting results. Online schema change tool, for frequent schema changes, especially adding indexes. This briefly locks the table, to setup triggers to track changes, copy data to a new table with the new desired schema, replay changes to the new table, then briefly lock the table again as you rename the new table as the target table. Manageability is a work in progress -- working on: - make InnoDB compression work for OLTP - Faker - tool for prefetching for replication slaves - replay workload is: page read, do some modification, page write. bottleneck might be disk reads, work is done by a single thread, transactions on master are concurrent. Faker has multiple threads replay transactions in "fake-changes" mode, no undo, no rollback, read-only, fetches into the buffer pool the pages needed for that transaction. Captures about 70% of disk reads for replication, they're working on fixes to get it up to 80-90%. - auto replacement - replace failed and unhealthy MySQL servers. - Auto resharding - sharding is easy, re-sharding is hard. open issues in manageability: diagnose why one host is slow, others are not. ....and some more.

Help Set the MariaDB 5.6 Roadmap with SkySQL and Monty Program

Fri, 03 Feb 2012 17:45:48 GMT

Part of every SkySQL subscription is paid to Monty Program to fund server development and improved functionality in MariaDB® (which is then sent upstream for inclusion in the MySQL® Server). Thus, we'd like to know what you'd like to see in the upcoming releases of the server. Please take a couple seconds (literally) and let your vote be heard: http://www.skysql.com/content/new-server-functionality-have-your-say We thank you and look forward to hearing your thoughts and ideas!

RMOUG Training Days 2012

Fri, 03 Feb 2012 17:45:00 GMT

The Rocky Mountain Oracle User Group ( RMOUG ) Training days are almost here. This year they are going to have a MySQL Oracle ACE Director, Ronald Bradford talk on MySQL Security Essentials. I will also be having a MySQL Crash Course . I was going to talk about replication but after attending the IOUG User summit and the seeing the vast interest in MySQL from Oracle users, I decided to take a step back and do this talk 1st. I have had a similar talk at a RMOUG meetup hands on labs in the past, but it has been updated.Alex Gorbachev of Pythian also has a session, Monitoring MySQL with Oracle Cloud Control 12c. Look forward to seeing everyone and I think great things are on the horizon with RMOUG and MySQL.

Game Over for NoSQL? Discussing Databases in Online Social Gaming

Fri, 03 Feb 2012 16:08:00 GMT

According to VentureBeat*, games companies raised a record-breaking $1.54 billion in funding last year and social gaming accounted for over half of that. No wonder everyone wants to have a piece of that pie!With the arrival of social network platforms, the gaming industry has seen an explosion in casual and social gaming. The social gamer represents a massive audience that cuts across all age, gender and demographic boundaries. Online social games are some of the most demanding applications in the world, with millions of users, stringent response times, complex simulation models and billing requirements. Games take years to develop for a reason ...Online social games are data-driven applications, and databases are central to these applications. However, there is no single database architecture that will fit the different types of data that the application needs to store. A data management architecture needs to account for the diversity of data, and optimize for some of the differences in the datatypes. E.g. it is ok to lose leaderboard data during a game as it can be reconstituted, whereas billing data needs to be 100% ACID. Therefore, with the generous contribution of Joshua Butcher, we just published a whitepaper that discusses the different types of data stored for various functions in social gaming. We will see that there cannot be a one-size-fits-all approach to database architecture, and suggest a sharding strategy based on schema partitioning. With our new whitepaper, we’re also starting a discussion on what the database of choice might be for anyone wanting to develop online social games. With so many NoSQL databases now available, one might wonder why MySQL would be a good database choice for the gaming industry. To find out, download our whitepaper today! If you have any questions or comments, feel free to reply to this blog below or reach out to us on Facebook, LinkedIn, Xing, Twitter or directly via these contact details.* http://venturebeat.com/2012/01/06/deanbeat-game-companies-raised-a-record-breaking-1-55b-in-2011/

Some guidelines for MySQL security | Nil Infobin [Digg]

Fri, 03 Feb 2012 12:23:07 GMT

Some guidelines for MySQL security

Event scheduler in MySQL 5.1 | Nil Infobin [Digg]

Fri, 03 Feb 2012 12:21:48 GMT

Event scheduler in MySQL 5.1

Log security and log tables.

Fri, 03 Feb 2012 10:55:07 GMT

Accidentially I came across the statement “SHOW GRANTS requires the SELECT privilege for the mysql database.” in MySQL documentation (http://dev.mysql.com/doc/refman/5.1/en/show-grants.html). It is not quite true. Any user can “SHOW GRANTS [FOR himself]” with no privileges at all. But more important: SELECT priviege is requried on database-level, Privilege to the privileges tables is not enough. See SHOW GRANTS; /*returns Grants for me@% —————————————————– GRANT USAGE ON *.* TO ‘me’@'%’ GRANT SELECT ON `mysql`.`user` TO ‘me’@'%’ GRANT SELECT ON `mysql`.`tables_priv` TO ‘me’@'%’ GRANT SELECT ON `mysql`.`procs_priv` TO ‘me’@'%’ GRANT SELECT ON `mysql`.`db` TO ‘me’@'%’ GRANT SELECT ON `mysql`.`columns_priv` TO ‘me’@'%’*/ SHOW GRANTS FOR root@localhost; /*returns Error Code: 1044 Access denied for user ‘me’@'%’ to database ‘mysql’ */ Anybody having SELECT privilege to the mysql database can read logs if you use log tables. And unlike when logging to files (where you can specify logfile paths) there is no option to specify another database for the log tables. I think it is a serious security flaw. The reason is that a log may contain data. That may be private data (email addresses, bank account numbers .. you name it). Consider a statement like UPDATE `identity`SET `bank_account_no` = ….. WHERE social_security_id = ….. ; You should have SELECT privilege to the `identity` table to see those data. But if you can read logs you need not. I don’t claim all applications would send such statements (but I believe that some will do). You can avoid it to some extent by using user variables, hashes, or by writing complex statements with JOINs and SUBQUERIES so that you don’t need to ‘ping-pong’ data from the server to the application and back (but it may conflict with performance in particular if SUBQUERIES are used). So you should be careful with logging to tables. Not only is performance not good, but there are security concerns. Who should be allowed to see the logs? (but still log tables may be convenient for a test/development scenario though.) I posted this bug report: http://bugs.mysql.com/bug.php?id=64215 Tweet

Event scheduler in MySQL 5.1

Fri, 03 Feb 2012 08:53:11 GMT

I just recently used Event scheduler which was the major feature in MySQL 5.1 version. Its very much similar to the linux crontab functionality. MySQL Event is nothing but a bunch of statements which we can run on specific schedule. … Continue reading →

SQL Server 2012 Integration Services - Overview

Fri, 03 Feb 2012 08:01:00 GMT

Marcin Policht presents a high-level overview of SQL Server 2012 Integration Services, covering some significant advances in the areas of usability, productivity, and manageability.

Log Buffer #257, A Carnival of the Vanities for DBAs

Fri, 03 Feb 2012 06:00:15 GMT

With new year many new projects, new technologies, new frameworks and new ideas are springing up at the speed of light and bloggers in the database arena are keeping up with this pace and this Log Buffer Edition is also living up to that pace and covers some of those posts in Log Buffer #257. [...]

STOP: DELETE IGNORE on Tables with Foreign Keys Can Break Replication

Fri, 03 Feb 2012 02:11:57 GMT

DELETE IGNORE suppresses errors and downgrades them as warnings, if you are not aware how IGNORE behaves on tables with FOREIGN KEYs, you could be in for a surprise. Let’s take a table with data as example, column c1 on table t2 references column c1 on table t1 – both columns have identical set of rows for [...]

Collaborate 2012 MySQL Sessions

Thu, 02 Feb 2012 20:50:07 GMT

Collaborate 2012 MySQL Sessions Please mark you calendars for the MySQL sessions at Collaborate this April in Las Vegas. Date Session ID Session Details Track Sun. Apr. 22 4:30 pm – 5:30 pm 9390 The Essentials of Data Discovery: Do you Know Where Your Data Is? Asset Lifecycle Management Mon. Apr. 23 9:45 am – 10:45 am 826 Virtualization Boot Camp: Virtualizing Oracle On VMware – Quick Tips Database Mon. Apr. 23 12:15 pm – 12:45 pm 554 Set Up MySQL In Five Minutes Flat (2-Part Session) Database Mon. Apr. 23 1:15 pm – 2:15 pm 578 Shell Scripting for MySQL Administration Database Mon. Apr. 23 2:30 pm – 3:30 pm 681 Portable SQL between Oracle and MySQL Development Mon. Apr. 23 3:45 pm – 4:45 pm 429 MySQL High Availability Solutions and case studies Database Mon. Apr. 23 5:00 pm – 6:00 pm 436 Making Sense of Parent-Child Relationships – The Oracle Kind, Not the Human Kind BI/Data Warehousing/EPM Mon. Apr. 23 5:00 pm – 6:00 pm 493 Administering MySQL for Oracle DBAs Database Mon. Apr. 23 5:00 pm – 6:00 pm 9050 Training? Where do I even begin? Professional Development Tue. Apr. 24 8:00 am – 9:00 am 281 Database Development Boot Camp: SQL Tips, Techniques and Traps to Avoid BI/Data Warehousing/EPM Tue. Apr. 24 8:00 am – 9:00 am 729 Challenges of Big Databases with MySQL Database Tue. Apr. 24 12:00 pm – 12:30 pm 389 MySQL for Oracle DBAs or How to Speak MySQL for Beginners (2-Part Session) Database Tue. Apr. 24 12:00 pm – 12:30 pm 1554 Set Up MySQL In Five Minutes Flat (2-Part Session) Database Tue. Apr. 24 2:00 pm – 3:00 pm 694 Security Boot Camp: Avoiding SQL Injection: Don’t Let a Stranger “Shoot You Up” Development Tue. Apr. 24 2:00 pm – 3:00 pm 809 Demystifying MySQL for Oracle DBAs and Developers Database Tue. Apr. 24 4:30 pm – 5:30 pm 362 Security Boot Camp: Security Around MySQL Database Wed. Apr. 25 12:00 pm – 12:30 pm 1389 MySQL for Oracle DBAs or How to Speak MySQL for Beginners (2-Part Session) Database Thu. Apr. 26 9:45 am – 10:45 am 9395 Understanding the ROI of Archiving: Why Data Archiving is a Must do for 2012 Application Strategy and Services

More details about SchoonerSQL performance, please!

Thu, 02 Feb 2012 20:37:52 GMT

Schooner has a blog post showing that one node of their product beats 9 nodes of Clustrix’s in throughput. But this reduces everything to a single number, and that’s not everything that matters. If you’ve looked at Vadim’s white paper about Clustrix’s (paid-for) performance evaluation with Percona, you see there is a lot of detail about how consistent the throughput and response time are. I’d love to see that level of details in any product comparison. A single number often isn’t enough to judge how good the performance is — fast is not the only thing that matters. I have absolutely no doubts that a single node of Schooner’s product can run like a deer. It isn’t doing any cross-node communication, after all, so it had better be faster than something that blends multiple nodes together into a virtual “single database server.” And I think if the full story were told, it would be a great knock-down drag-out fight. Give us more details, Schooner! Further Reading:High Performance MySQL, Second Edition: Query Performance Optimization An ongoing thread of blogs on MySQL performance Status update on High Performance MySQL High Performance MySQL Third Edition pre-order available Sessions of interest at the Percona Performance Conference

Collaborate 2012 Registration is Now Open!

Thu, 02 Feb 2012 18:11:32 GMT

http://collaborate12.ioug.org Double Down at COLLABORATE 12- The IOUG Forum with Two Ways to Save- and a Chance to Win! The user-driven Oracle event of the year is fast-approaching, and IOUG wants you to make youreducational experience a sure bet. Between hundreds of cutting-edge education sessions, workshops and legendary Oracle speakers, you’ll return from Las Vegas with valuable knowledge to transform into immediate results for your business. No need to go all in to attend- IOUG is sweetening the pot with ways for you to save big bucks and even pocket some cash while you’re at it. Register today for your chance to win a $200 American Express Gift Card!The deck is stacked at COLLABORATE 12 – The IOUG Forum in your favor. Register for the conference today with the priority code EM03 and be immediately entered for a chance to add a $200 AmEx Gift Card to your winnings for the week. Treat yourself to some fusion cuisine while in Vegas, catch a show on the strip, or just pocket the prize for a rainy day! Buy-in to the table at COLLABORATE 12-The IOUG Forum has never been more reasonable. Hotel rates are only going up from here on out, so book your room today to take advantage of the final day of ultra-reasonable rates! How about adding more savings to the mix on your conference registration?Sign up for a seat at the table with IOUG before February 29 and be dealt a winning hand: save up to $500 on conference registration by booking ahead of the early bird, and we’ll even throw in a bonus, extra day of education($599 value)! Don’t miss your chance to to take advantage of these massive savings . Book your COLLABORATE 12 journey today! Guarantee your seat at the table today, and save by registering before the Early Bird Deadline! Registration rates start as low as $1,295* for members of IOUG. Resolve to give yourself a fantastic week of Oracle Education, and happy 2012! * Rate assumes hotel and group discount Boot Camp Information Housing Deep Dive Information IOUG Registration Benefits Exhibit at COLLABORATE 12 Session Schedule Want to sponsor a Deep Dive? Contact Jconlon@ioug.org for more information.

Last chance to take part in our MySQL/NoSQL/NewSQL survey

Thu, 02 Feb 2012 17:13:10 GMT

Thanks to everyone who has already taken part in our survey exploring changing attitudes to MySQL following its acquisition by Oracle and examining the competitive dynamic between MySQL and other database technologies, including NoSQL and NewSQL. The response has been great and even a quick look at the results makes for interesting reading, particularly in the light of our previous findings which indicated declining MySQL usage. I am really looking forward to having the opportunity for a deep dive into the results and break out the figures to get a better understanding of the potential impact of alternative MySQL distribution and support providers, as well as NoSQL and NewSQL, on continued usage of MySQL. The survey results will be made freely available on our blogs, as well as being included in a long format report containing our additional analysis and research related to the MySQL ecosystem and competitive dynamic. Right now, however, is your last chance to contribute to the survey and get your voice heard. There are just 12 questions to answer, spread over four pages, and the entire survey should take no longer than five minutes to complete. All individual responses are of course confidential. The survey will close in 24 hours.

Resolving the bison.exe m4 Invalid argument Error when building MySQL/MariaDB/XtraDB on Windows

Thu, 02 Feb 2012 15:46:46 GMT

If you’ve tried compiling MySQL/MariaDB/XtraDB from source code on Windows, you may have run into the following error: C:\GnuWin32\bin\bison.exe: m4: Invalid argument Now, it is a known bug to receive this error if you install it to a location with spaces in the path. So don’t do that! But, there are cases when you have not installed it to a path with spaces, and you still receive this error. I’ve encountered it a number of times, and I’m not the only one (btw, thanks Venu for your existing work-around up to this point!). … Problem: Compile MySQL/MariaDB/XtraDB on Windows and receive the following error during compilation: C:\GnuWin32\bin\bison.exe: m4: Invalid argument Troubleshooting: As you can see, there is no space in the path that is even reported. Let’s double-check that that is the path returned from the command line: C:\mariadb-5.3\bld>where bison C:\GnuWin32\bin\bison.exe C:\mariadb-5.3\bld>where m4 C:\GnuWin32\bin\m4.exe Again, both are stored in a location without spaces. Let’s double-check what CMake reports for the bison path in CMakeCache.txt: //path to the bison executable BISON_EXECUTABLE:FILEPATH=C:/GnuWin32/bin/bison.exe Again, the same path with no spaces. In the build output (either from Visual Studio or the Command Line), near the “bison.exe: m4: Invalid argument” error, you should see a link to the “build log” for that which failed, which contains additional information. Opening that file, you’ll see this is the full command that is trying to be run: C:\GnuWin32\bin\bison.exe -y -p MYSQL --output=C:/mariadb-5.3/bld/sql/sql_yacc.cc --defines=C:/mariadb-5.3/bld/sql/sql_yacc.h C:/mariadb-5.3/sql/sql_yacc.yy I recommend trying to run it from the command line yourself. If you’re already seeing the “bison.exe: m4: Invalid argument” error, then you’re likely going to see it here (which is good for later testing): C:\mariadb-5.3\bld>C:\GnuWin32\bin\bison.exe -y -p MYSQL --output=C:/mariadb-5.3/bld/sql/sql_yacc.cc --defines=C:/mariadb-5.3/bld/sql/sql_yacc.h C:/mariadb-5.3/sql/sql_yacc.yy C:\GnuWin32\bin\bison.exe: m4: Invalid argument So what is going on? Well, thanks to Wlad(!), he recommended I run Procmon to see what it reports. I downloaded and ran procmon.exe. Run bison command from command line: C:\GnuWin32\bin\bison.exe -y -p MYSQL --output=C:/mariadb-5.3/bld/sql/sql_yacc.cc --defines=C:/mariadb-5.3/bld/sql/sql_yacc.h C:/mariadb-5.3/sql/sql_yacc.yy In procmon, search (ctrl + f) for “bison” Locate an entry (may see many) and look at the paths for the bison.exe executable (it is easy output to read). The $Path for the bison entry said: C:\ProgramData\Microsoft\Windows\Start Menu\Programs\GnuWin32\Bison Aha! For one, that is just a directory with no executables. But for two, and more importantly, it has a “space” in the path. So, that is the root cause of the problem. Solution: Initially, I had run an installer to install GnuWin32 Bison. This added docs shortcuts to the C:\ProgramData\Microsoft\Windows\Start Menu\Programs\GnuWin32\Bison location (and apparently created some registry entry that ultimately was used by VS2008), and links to those docs in my Start Menu. Personally, I know where the docs are, so I can live without them being in my “Start Menu” (heck, maybe you could even copy them back, but it ultimately needs removed from the registry). So, if you’re the same way, then this should work for you: Copy C:\GnuWin32 (and subdirs) to a safe location Run the bison uninstall (to remove the entry from the registry) Ensure current C:\GnuWin32 does not exist Ensure C:\ProgramData\Microsoft\Windows\Start Menu\Programs\GnuWin32 does not exist (those true docs will already be saved in the copy we made anyway, that will be restored to C:\GnuWin32) Re-boot computer (because stale registry values will remain and thus a restart is necessary – this I also detected with Procmon) Copy the saved GnuWin32 (and subdirs) back to C:\. Ensure C:\GnuWin32\bin is in the $PATH for Windows. Now, re-run your command from the command line (or rebuild – but the command line command is much quicker for testing): C:\mariadb-5.3\bld>C:\GnuWin32\bin\bison.exe -y -p MYSQL --output=C:/mariadb-5.3/bld/sql/sql_yacc.cc --defines=C:/mariadb-5.3/bld/sql/sql_yacc.h C:/mariadb-5.3/sql/sql_yacc.yy C:\mariadb-5.3\bld> \o/ Once that’s good to go, you’ll know your build will run successfully too (at least it won’t fail with this dreaded m4 error.) Hope this helps.

MySQL Conference and Expo Talk on Benchmarking

Thu, 02 Feb 2012 15:28:23 GMT

I’ll be speaking on April 11th at 4:30 pm in Room 4 in at the Percona Conference and Expo Talk. The topic will be “Creating a Benchmark Infrastructure That Just Works.” Throughout my career I’ve been involved with maintaining the performance of database applications and therefore created many benchmark frameworks. At Tokutek, an important part of my role is measuring the performance of our storage engine over time and versus competing solutions. There is nothing proprietary about what I’ve created, it can be used anywhere. My presentation will cover how I created the benchmark infrastructure at Tokutek: Hardware and software considerations (including physical vs. virtual) Selecting benchmarks Capturing detailed information during the benchmark Automation Storing results Visualization Trend analysis Continuous integration (monitoring the performance of future versions) Self-service (let people get the information they want) Track: Tools Experience level: Intermediate Tokutek is also a sponsor of the show and will have an expo booth. So, I hope to see you at my talk and/or at our booth.

MySQL[plus] awards 2011 final results ! (and a story)

Thu, 02 Feb 2012 15:00:07 GMT

Let me tell you a very touching story about these awards. As I said in the comments : Sometimes we think about something and the next minute we did it… And it exactly what happened, I took my inspiration from the TUAW Best Of 2011 (Yes, I’m also an Apple fan boy, sorry :-)), and a few minutes later, MySQL[plus] awards 2011 were live ! I thought I could have 50 voters, perhaps 100 voters, in my wildest dreams… But more than 300 voters and more than 4000 views later, the reality is there : You are amazing ! But, no, it wasn’t perfect ! Shlomi said “You’ve got yourself into a bottomless pit!”, and it’s true. Ronald said “Your options leave a lot out”, and it’s true. But I tried to offer this overview of the most used tools with spontaneity, bravery and honesty. And in my heart, I know you enjoyed it (no ?) It now remains 11 months to prepare the coming awards… So, final results are available : http://www.mysqlplus.net/2012/01/05/vote-mysqlplus-community-awards-2011 Remember that the real winner is the community ! PS : When I said “Answer with your heart first”, this meant that there was no place for cheaters… Related Posts :Does mpstat can replace vmstat ?Vote for MySQL[plus] awards 2011 !How to quickly identify queries with pt-query-digest and pt-query-advisor from rules ?Monitoring your monitoring tools (MONyog inside) ![UPDATED] Why you should go to Percona Live (London) ?

One SchoonerSQL node is equavalent to 9 Clustrix nodes

Thu, 02 Feb 2012 01:44:00 GMT

Percona recently evaluated the performance of Clustrix appliances by running tpcc-mysql benchmark at a scale factor of 5000 warehouse. Based on Percona’s blog, each Clustrix appliance node comprises of the following:2x Intel 4-core processors (Xeon)48GB RAM (40GB allocated to InnoDB buffer pool)7x Intel SSD G2 (160GB each), software RAID0The following results are taken from the report for Clustrix (tpcc-mysql new-order transactions/ 10 seconds):For convenience, the above results have been converted to TPM (transactions/minute):You can download the detailed white paper here: Clustrix_TPCC_Percona_White PaperThe SchoonerSQL ComparisonWe were curious to see what a single instance of SchoonerSQL (a full distribution of MySQL/InnoDB) on a single node could compare at the same benchmark scale - 5000 warehouses with the same benchmark, tpcc-mysql from Percona.SchoonerSQL was installed on a system with the following configuration:2x Intel 6-core processors (Westmere)92GB DRAM (72GB used for InnoDB buffer pool)8x OCZ SSDs (200GB each) (can be alternatively replaced with FusionIO or Intel)2x SAS HDDs with NVRAM in RAID controller (for OS and commit log) Result summary: At 64 connections with exactly the same workload conditions, a single SchoonerSQL node measured more throughput than 9 Clustrix nodes.

Basic ETL with Gearman and MySQL in a few lines of PHP code

Wed, 01 Feb 2012 23:12:00 GMT

Gearman is awesome. If you do not know what it is, its a queue and load balancing system for an arbitrary number of workers which enables distributed computing across many nodes. Some of the same guys who worked on mySQL source worked on Gearman.Feel free to search my blog on other gearman uses.The Problem:We store a lot of stats, make a lot of changes and we want to see the result of the stats in realtime. Our stat system is pretty slick. For each tag increment the application increments a count and group said tag by minute, hour, month with a hash tag numeric representation of the text for compact writes. This means 1 tag write produces 4 SQL statements. We track over 239211 distinct tags at around 10K Writes per second on a single mySQL instance on EC2 in a RAID-10 EBS xtra-large Config (although because EBS mirrors internally I can just raid-0 but I was too scared).Once the mySQL instance hits disk (EBS) our throughput becomes very unstable, possibly slowing down the site.The solution was to defer these writes but how can I do it without building a logging system, aggregator, loader and having a bunch of moving parts? Really I want to only spend 10 mins on this problem and use existing monitoring code. So my 10 min solution: 3 mins to write the code 6 mins testing 55 seconds of patting myself on back 5 seconds to deploySolution Detail:Since Gearman workers connect to the GearmanD server's Job QUEUE and loop for more Jobs. This means program stays in memory for the length of the process (until worker restart). This means I can transform the data in application memory. Since the program is persistently connected to the DB that means I can periodically load the data in chunks.In stead of having 100s of possible concurrent connections doing writes I can control the writes based on the number of workers. Innodb is very fast and consistent at low levels of concurrency (less then 50).Since I can drain the queue from GearmanD and transform the data locally I do not really need to worry about running out of memory on the queue server. The consumer is faster then the producer. I can combine 1000s of writes into a single write.Let's look at some code: self::MAX_NUMBER_OF_EVENTS){ return self::load(); } // // flush if its our time // if (self::$nextWrite < time()){ return self::load(); } return; } /* * flush the stored tags to the database * @returns void */ protected static function load() { // // write transformed events to the db // $thisRun = 0; foreach(self::$eventTable as $event => $sum) { EventTrackerDB::singleton()->updateEvent($event, $sum); $thisRun++; self::$totalEvents++; } $msg = "EventTracker write complete $thisRun events this run and a total of " . self::$totalEvents . " events written so far"; Debugger::log("OT", $msg); // // re-init // self::$nextWrite = time() + self::FLUSH_INTERVAL; self::$eventTable = array(); } }?>In summary with gearman I am able to process 250K events in seconds. The queue never builds up and there is special code to handle kills (not SIGKILL).

DB2 z/OS Systems Tuning Tactics

Wed, 01 Feb 2012 08:01:00 GMT

In the z/OS
environment, tuning a DB2 subsystem may require the DBA and systems programmers
to wear multiple hats and approach overall performance tuning from several
different angles. What approaches are best in which situations?

Usage and Benefits of Using WITH RESULT SETS In SQL Server 2012

Mon, 30 Jan 2012 08:01:00 GMT

Having stored procedures returning result sets can become a problem when you want to have control of the name and data type of the columns being returned. Arshad Ali explains a new SQL Server 2012 feature, which lets you redefine the name and data types of the columns being returned from the stored procedure.

Managing Data Growth with Enterprise-class Databases

Fri, 27 Jan 2012 08:01:00 GMT

Compliance isn’t the only reason to retain large data sets; better historical business reference and real-time analysis, all require access to large volumes of data to gain better business insights. Read on to learn more.

SQL Server 2008 and 2008 R2 Integration Services - Sort Transformation

Wed, 25 Jan 2012 08:01:00 GMT

SQL Server 2008 R2 Integration Services offer a wide range of pre-built components, which deliver generic functionality commonly required when performing ETL tasks. While most of them are fairly straightforward to deploy in their basic form, there are caveats that should be taken into account as part of their implementation. Read on to learn more.

Data Driven Subscription in SSRS 2008 R2

Mon, 23 Jan 2012 08:01:00 GMT

Learn how to acquire data or parameter values required during the execution of a report from a database query, using Data-Driven Subscription in SQL Server Reporting Services (SSRS).

Couchbase Moving Forward In 2012

Fri, 20 Jan 2012 22:18:00 GMT

NoSQL startup is gearing up for a major new database release.

SQL Server 2008 and 2008 R2 Integration Services - Sampling Transformations

Fri, 20 Jan 2012 08:01:00 GMT

Row Sampling Transformation and Percentage Sampling Transformation, which provide data sampling functionality, are useful in a number of Analysis Services and Data Mining-based scenarios. Marcin Policht demonstrates the functionality of both transformations.

Report Subscription in SSRS 2008 R2

Wed, 18 Jan 2012 08:01:00 GMT

SSRS allows you to create two types of subscriptions, e.g., Standard Subscription and Data Driven Subscription. This article discusses the standard subscription in which you need to specify the parameters the report expects at the time of subscription creation and which cannot be changed at runtime.

SQL Server 2008 and 2008 R2 Integration Services - Dimension Processing Data Flow Destination

Fri, 13 Jan 2012 08:01:00 GMT

The SSIS component, Partition Processing Data Flow Destination, is capable of performing direct data load from a fact table into an arbitrarily chosen partition of a measure within an Analysis Services cube. In this article Marcin Policht examines its counterpart known as Dimension Processing Data Flow Destination, which provides equivalent functionality in regard to dimensions.

Archiving Oracle

Wed, 11 Jan 2012 08:01:00 GMT

Archiving older data is a complex task. Within the context of an Oracle database there are methods of archiving data, some simple, some a bit more complex but still within the realm of possibility. Let's look at those options and what they can, and cannot, offer.

Understanding SSRS Report Definition Language (RDL)

Mon, 09 Jan 2012 08:01:00 GMT

SSRS reports are defined in Report Definition Language (RDL), which is nothing but Extensible Markup Language (XML) adhering to a defined report definition language schema. This article examines RDL and some of the elements that you will notice in your report's RDL.

The Format() Function in SQL Server 2012

Fri, 06 Jan 2012 08:01:00 GMT

The only formatting capabilities that SQL Server has had until now were the CAST and CONVERT function, forcing developers to write user defined functions using native string functions/date functions or CLR. The long wait for this format function is over; SQL Server 2012 RC0 is leveraging the .NET format capabilities by introducing the FORMAT() function.

SQL Server 2008 and 2008 R2 Integration Services - Partition Processing Data Flow Destination

Wed, 04 Jan 2012 08:01:00 GMT

The primary purpose of Integration Services technology is to deliver ETL functionality that supports core features of SQL Server database management systems. However, many times its role involves interaction with components that provide Business Intelligence-related features. This article discusses one such scenario, which leverages built-in SSIS Partition Processing Data Flow Destination in order to facilitate Analysis Services processing.

Setting Up Email Notification for SSIS Package Failure

Fri, 30 Dec 2011 08:01:00 GMT

Learn how to enable the logging functionality for SQL Server Integration Services (SSIS) and how to capture detailed information for immediate troubleshooting without re-running the package.

Report Snapshots in SSRS 2008 R2

Wed, 28 Dec 2011 08:01:00 GMT

Arshad Ali explores the report snapshot feature in SQL Server Reporting Services (SSRS). Learn how to show data at a point of time or how you can subscribe to your published reports.

SQL Server 2008 and 2008 R2 Integration Services - Customizing Slowly Changing Dimension Transformation

Fri, 23 Dec 2011 08:01:00 GMT

While SQL Server 2008 R2 Integration Services' Changing Attribute and Historical Attribute options facilitate a variety of different data analysis scenarios, its wizard-driven configuration is relatively inflexible and generates a Data Flow structure whose performance tends to be suboptimal. This article presents a few methods of remediating these shortcomings.

Physical Database Design Consideration

Wed, 21 Dec 2011 08:01:00 GMT

Greg Larsen explores different physical database design elements that should help you make the right design choice for your database.

Can you Make Money with a Cloud Database as a Service?

Mon, 19 Dec 2011 23:45:00 GMT

Xeround has raised millions for its database app technology as demand for cloud services ramps up.

Operational Scalability and Security with Sybase Adaptive Server Enterprise

Mon, 19 Dec 2011 08:01:00 GMT

Robert Schneider defines the elements that comprise operational scalability, and then provide concrete examples of how Sybase Adaptive Server Enterprise (ASE) augments operational scalability via minimizing downtime, sustaining heavy workloads, and supplying enterprise-grade security.

Report Caching in SQL Server Reporting Services 2008 R2

Fri, 16 Dec 2011 08:01:00 GMT

To improve the performance of report processing, SQL Server Reporting Services (SSRS) lets you enable caching for the report so that if the same report request comes again, the stored copy can be rendered in the desired format and served instead of processing it from scratch. This article explores this caching feature in detail.

SQL Server 2008 and 2008 R2 Integration Services - Slowly Changing Dimension Transformation - Historical Attributes

Wed, 14 Dec 2011 14:00:00 GMT

Recently we described the characteristics of Slowly Changing Dimension methodology, focusing in particular on its implementation in SQL Server 2008 R2 Integration Services. Its wizard-driven transformation does not offer full support for all existing change types. We continue our coverage of this subject by presenting an equivalent setup that eliminates this shortcoming.

New Date,Logical Functions in SQL Server 2012

Mon, 12 Dec 2011 14:00:00 GMT

With the official release of SQL Server 2012 RC0, Microsoft has added more features and enhancements to Transaction SQL. This article illustrate some of the new the DATE and LOGICAL functions introduced in SQL Server 2012.

Patching a Newly Added Node for an Existing SQL 2005 Clustered Instance

Fri, 09 Dec 2011 13:00:00 GMT

Learn how to apply a service pack and hotfix to a newly added SQL Server 2005 clustered node.

Creating and using Report Parts in Report Builder 3.0

Wed, 07 Dec 2011 14:00:00 GMT

Arshad Ali shows you how to create and publish a report part in SQL Server 2008 R2 using Report Builder 3.0.

SQL Server 2008 and 2008 R2 Integration Services - Slowly Changing Dimension Transformation - Type 1 Changes

Mon, 05 Dec 2011 15:00:00 GMT

An introductory article into the subject of Slowly Changing Dimension from the perspective of SQL Server 2008 R2 Integration Services.

Oracle 11gR2 I/O Performance Tuning: Using Solid State Drives

Fri, 02 Dec 2011 16:39:00 GMT

Solid-state drives (SSDs) can make an incredible difference when applied judiciously to an apparently intractable I/O performance bottleneck.

Salesforce Heroku Offers Standalone Cloud-Based PostgreSQL Database

Mon, 28 Nov 2011 19:43:00 GMT

Since 2007, Heroku has offered database-as-a-service to its platform-as-a-service customers for use with Heroku platform applications. Now the company is extending the service for use with any cloud platform.

Achieving Five Nines Availability Using Synchronous Cluster Replication for MySQL

Wed, 23 Nov 2011 20:57:00 GMT

SchoonerSQL can help you achieve five nines availability for your MySQL database.

Thoughts on Improving DBA Productivity

Fri, 18 Nov 2011 22:57:00 GMT

Lockwood Lyon offers ideas for how database administrators can react less, think more and find lasting solutions.

Sybase ASE Application Development and Big Data

Tue, 15 Nov 2011 12:59:00 GMT

Sybase ASE can now work with unstructured data and very large data volumes.

Data Mining Model Training Destination in SQL Server Integration Services

Thu, 10 Nov 2011 20:29:00 GMT

How to incorporate data mining directly into the Data Flow of SQL Server 2008 R2 Integration Services-based packages.

MongoDB - Why Does NoSQL Matter?

Mon, 07 Nov 2011 23:31:00 GMT

NoSQL isn't a fad, it's delivering real business value today.

Using Autoextend to Increase Oracle Database File Sizes

Fri, 04 Nov 2011 20:59:00 GMT

Autoextend can save DBAs precious time by allowing Oracle to automatically increase database file sizes to a pre-determined limit.

Finding the Source of Your SQL Server Database I/O

Tue, 01 Nov 2011 00:59:00 GMT

There are a number of ways of measuring and identifying I/O-related performance information for your SQL Server database instances.

Database Buyer's Guide Conclusion: Which Database Is Best for You?

Thu, 27 Oct 2011 10:57:00 GMT

Desktop databases, server databases, the Cloud or open source – which route is the right one for your organization?

Neo Launches NoSQL Graph Database

Tue, 25 Oct 2011 21:29:00 GMT

New type of 'graph' based NoSQL database debuts with tight integration for Spring-based development.

Why You'll Want to Upgrade to Sybase ASE 15.7

Fri, 21 Oct 2011 19:29:00 GMT

Sybase ASE 15.7 has a bunch of new features you may want to consider, including improved performance, scalability and security.

Getting Started with SQL Server Report Builder 3.0

Tue, 18 Oct 2011 20:59:00 GMT

SQL Server offers a simple tool for business users to perform ad-hoc reporting.

Data Mining Query Transformation in SQL Server Integration Services

Fri, 14 Oct 2011 22:59:00 GMT

We look at another way to execute Data Mining Extension (DMX) queries against data mining models in SSIS.

Microsoft 'Denali' Set for 2012 Delivery

Wed, 12 Oct 2011 17:59:00 GMT

At Microsoft's premier event for SQL Server developers, the company renames "Denali" and gives glimpses of its future.

Cloud Database Buying Guide

Tue, 11 Oct 2011 10:58:00 GMT

Cloud databases can save you time and money, but use them carefully.

Color Coding Your SQL Server Connections

Fri, 07 Oct 2011 21:27:00 GMT

Here's an easy way to keep track of your SQL Server instances.

An Easy Way to Monitor SQL Server Replication

Wed, 05 Oct 2011 19:29:00 GMT

SQL Server replication is important for keeping data consistent. Here's how to make sure it's working properly.

Oracle Goes Big for NoSQL

Tue, 04 Oct 2011 18:59:00 GMT

Oracle explains what 'big data' is all about and delivers new Hadoop, R and NoSQL database hardware and software.

Getting Started with Utility Control Point in SQL Server 2008 R2

Wed, 28 Sep 2011 21:57:00 GMT

Utility Control Point can help SQL Server DBAs in multi-server administration.

Microsoft SQL Server 2008 R2 spotlight

Thu, 20 May 2010 15:54:58 GMT

Microsoft's latest database server release comes with a host of new features, including several BI enhancements for the enterprise.

Microsoft SQL Server 2008 R2 spotlight

Thu, 20 May 2010 15:54:58 GMT

Microsoft's latest database server release comes with a host of new features, including several BI enhancements for the enterprise.

Seizing stats and grouped data with T-SQL aggregate functions

Wed, 19 May 2010 03:00:00 GMT

The power of aggregate functions goes well beyond the basics. This tip demonstrates how to work with grouped data, verify checksums, and retrieve detailed statistics.