MySQL Query Profiling with Performance Schema

May 26th, 2015 Alex Gorbachev No comments

One of my favorite tools for query optimization is profiling. But recently I noticed this warning:

mysql> set profiling=1;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> show warnings;
| Level   | Code | Message                                                              |
| Warning | 1287 | '@@profiling' is deprecated and will be removed in a future release. |

After looking through certain documentation , I should indeed start using the Performance Schema to get this information.

Okay, so let’s give that a try.

I confirmed that I started MySQL 5.6.23 with the default of Performance Schema = ON:

mysql> show global variables like '%perf%';
| Variable_name                                          | Value |
| performance_schema                                     | ON    |

I’ll be using a development server for doing query profiling, so I can turn all of these on:

mysql> update performance_schema.setup_instruments set enabled='YES', timed='YES'; #you want the stage* ones enabled
mysql> update performance_schema.setup_consumers set enabled='YES'; #you want the events_statements_history* and events_stages_history* enabled

Start with fresh collection tables:

mysql> truncate performance_schema.events_stages_history_long;
mysql> truncate performance_schema.events_statements_history_long;

Then turn the profiler on:

mysql> set profiling=1;

Now run a sample query:

mysql> select distinct(msa) from;

And find the resulting event IDs to use in the query below:

mysql> select event_id, end_event_id, sql_text from performance_schema.events_statements_history_long where sql_text like '%msa%';
|      41 |       938507 | select distinct(msa) from                                                                  |

Insert those beginning and ending event IDs, and here’s the new profiling output on my test query from Performance Schema:

mysql> select substring_index(event_name,'/',-1) as Status, truncate((timer_end-timer_start)/1000000000000,6) as Duration from performance_schema.events_stages_history_long where event_id>=41 and event_id<=938507;

| Status               | Duration |
| init                 | 0.000103 |
| checking permissions | 0.000006 |
| Opening tables       | 0.000051 |
| init                 | 0.000014 |
| System lock          | 0.000007 |
| optimizing           | 0.000003 |
| statistics           | 0.000011 |
| preparing            | 0.000011 |
| Creating tmp table   | 0.000048 |
| executing            | 0.000002 |
| Sending data         | 1.251331 |
| end                  | 0.000003 |
| removing tmp table   | 0.000008 |
| query end            | 0.000006 |
| closing tables       | 0.000009 |
| freeing items        | 0.000111 |
| cleaning up          | 0.000002 |

Compare the legacy profiling available for the query:

mysql> show profile for query 1;
| Status               | Duration |
| starting             | 0.000125 |
| checking permissions | 0.000007 |
| Opening tables       | 0.000020 |
| init                 | 0.000014 |
| System lock          | 0.000007 |
| optimizing           | 0.000003 |
| statistics           | 0.000011 |
| preparing            | 0.000011 |
| Creating tmp table   | 0.000027 |
| executing            | 0.000001 |
| Sending data         | 1.353825 |
| end                  | 0.000005 |
| removing tmp table   | 0.000007 |
| end                  | 0.000002 |
| query end            | 0.000006 |
| closing tables       | 0.000009 |
| freeing items        | 0.000069 |
| cleaning up          | 0.000028 |

The obvious question is: Why I would want to be limited to this information when the Performance Schema has so much more available?

But this proves we can get profiler information in a format we’re used to when MySQL fully deprecates the profiling tool.


Learn more about Pythian’s expertise in MySQL.

Categories: Alex @ Pythian Tags:

Making Existing SQLPLUS Scripts 12c and Container DB (PDB) Compatible

May 26th, 2015 Alex Gorbachev No comments

Oracle 12c introduces new catalog features including CDB_ dictionary views (which include a CON_ID column) superseding the DBA_ views that most DBA sqlplus scripts are based upon.

However, existing DBA sqlplus scripts can easily be modified using just a few simple sqlplus techniques to be compatible with 11g, as well as all types of 12c databases including legacy and container databases.

The following simple SQL and sqlplus techniques can be used to make a “universal script” that is compatible with all versions.

Illustrating the Issue

Let’s say for sake of example that we have a simple 10g/11g monitoring script that’s checking the amount of freespace in each tablespace by querying the DBA_TABLESPACE_USAGE_METRICS view.

On our 10g or 11g database the following query gives the necessary information:

SQL> select version from v$instance;


SQL> select tablespace_name, tablespace_size, used_percent
  3  order by tablespace_name;

------------------------------ --------------- ------------
FCCDEV                                  256000      .053125
SYSAUX                                 1024000   31.0617188
SYSTEM                                 1024000   9.19453125
TEMP                                   1024000            0
UNDOTBS1                               1024000      .015625
USERS                                   256000        1.275

6 rows selected.



Now will the same query work on a 12c database? Of course it will:

SQL> select version from v$instance;


SQL> select tablespace_name, tablespace_size, used_percent
  3  order by tablespace_name;

------------------------------ --------------- ------------
SYSAUX                                 4194302   .773048769
SYSTEM                                 4194302   1.05991414
TEMP                                   4194302            0
UNDOTBS1                               4194302   .031280532
USERS                                  4194302   .003051759



It executes successfully on the 12c database but there’s a problem: the query is only returning the data from the root container (or more accurately, from the container in which the statement was executed). The PDB data is missing, I have both open and closed PDBs in this database:

SQL> select con_id, name, open_mode from V$CONTAINERS order by con_id;

    CON_ID NAME                           OPEN_MODE
---------- ------------------------------ ----------
         1 CDB$ROOT                       READ WRITE
         2 PDB$SEED                       READ ONLY
         3 TEST1                          READ WRITE
         4 LDB3                           MOUNTED



The LDB3 PDB is closed (mounted) so I’m not interested in monitoring the tablespace freespace in it but I am interested in the details from the opened TEST1 PDB.

To get the required information we need to make two or three (Third being optional) changes:

1) Change the view from DBA_ to CDB_
2) Add the CON_ID column to the output
3) Add the CON_ID column to the ORDER BY clause

Hence (executing from CDB$ROOT) the query becomes:

SQL> select con_id, tablespace_name, tablespace_size, used_percent
  3  order by con_id, tablespace_name;

---------- ------------------------------ --------------- ------------
         1 SYSAUX                                 4194302   .773048769
         1 SYSTEM                                 4194302   1.05991414
         1 TEMP                                   4194302            0
         1 UNDOTBS1                               4194302   .031280532
         1 USERS                                  4194302   .003051759
         3 AUDIT_DATA                               64000        .2875
         3 SYSAUX                                 4194302   .410843091
         3 SYSTEM                                 4194302   .474167096
         3 TPCCTAB                                1024000   5.63203125

9 rows selected.



So that works fine, but as it stands we have two versions of the query and therefore we need two monitoring scripts.


Building Blocks for the Universal Script

Applying a number of simple sqlplus techniques can help us with this and will allow us to make the single universal version of the sqlplus script.

1) Use a SQLPLUS variable:

The sqlplus DEFINE command allows us to define variables. We can easily define a variable that tells us which view prefix to use depending on whether the database version is 11g or 12c.

SQL> COLUMN view_prefix NEW_VALUE view_prefix
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','CDB','DBA') view_prefix FROM v$instance;




2) Dynamically build the view name:

The second tip is that in sqlplus to concatenate a variable with a string a period must be used to show where the variable name ends:

SQL> prompt &view_prefix

SQL> prompt &view_prefix._TABLESPACE_USAGE_METRICS



Plugging that into the original query gives:

SQL> select tablespace_name, tablespace_size, used_percent
  2  from &view_prefix._TABLESPACE_USAGE_METRICS
  3  order by tablespace_name;
old   2: from &view_prefix._TABLESPACE_USAGE_METRICS

------------------------------ --------------- ------------
AUDIT_DATA                               64000        .2875
SYSAUX                                 4194302   .410843091
SYSAUX                                 4194302   .773048769
SYSTEM                                 4194302   1.05991414
SYSTEM                                 4194302   .474167096
TEMP                                   4194302            0
TPCCTAB                                1024000   5.63203125
UNDOTBS1                               4194302   .031280532
USERS                                  4194302   .003051759

9 rows selected.


But we’re missing the container ID column.


3) Add columns dynamically using additional sqlplus variables:

We can “optionally” include columns such as the CON_ID column using the same technique:

SQL> COLUMN view_prefix NEW_VALUE view_prefix NOPRINT
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','CDB','DBA') view_prefix FROM v$instance;

SQL> COLUMN con_id_col NEW_VALUE con_id_col NOPRINT
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','con_id,','') con_id_col FROM v$instance;

SQL> select &con_id_col tablespace_name, tablespace_size, used_percent
  2  from &view_prefix._TABLESPACE_USAGE_METRICS
  3  order by &con_id_col tablespace_name;
old   1: select &con_id_col tablespace_name, tablespace_size, used_percent
new   1: select con_id, tablespace_name, tablespace_size, used_percent
old   2: from &view_prefix._TABLESPACE_USAGE_METRICS
old   3: order by &con_id_col tablespace_name
new   3: order by con_id, tablespace_name

---------- ------------------------------ --------------- ------------
         1 SYSAUX                                 4194302   .773239504
         1 SYSTEM                                 4194302   1.05991414
         1 TEMP                                   4194302            0
         1 UNDOTBS1                               4194302   .003814699
         1 USERS                                  4194302   .003051759
         3 AUDIT_DATA                               64000        .2875
         3 SYSAUX                                 4194302   .410843091
         3 SYSTEM                                 4194302   .474167096
         3 TPCCTAB                                1024000   5.63203125

9 rows selected.



Note that the comma is in the variable and not in the column list in the SQL SELECT or ORDER BY clauses.

The script is now dynamically determining whether to use the CDB_ or DBA_ view and similarly dynamically adding the CON_ID column to the SELECT and ORDER BY clauses. (And of course should be executed from the root container.)

And the exact same script still works on the 11g database using the 11g version of sqlplus!

Similarly the optional column (including the comma) defined in the sqlplus variable could be used in an aggregation GROUP BY clause. However, if the query has no other aggregation columns then we might need to add a constant to the GROUP BY clause (and ORDER BY), otherwise the GROUP BY would have no columns listed and the universal sqlplus script is executed against an 11g database.

For example:

SQL> COLUMN view_prefix NEW_VALUE view_prefix NOPRINT
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','CDB','DBA') view_prefix FROM v$instance;

SQL> COLUMN con_id_col NEW_VALUE con_id_col NOPRINT
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','con_id,','') con_id_col FROM v$instance;

SQL> select &con_id_col min(extended_timestamp), max(extended_timestamp)
  2  from &view_prefix._AUDIT_TRAIL
  3  group by &con_id_col 1 order by &con_id_col 1;
old   1: select &con_id_col min(extended_timestamp), max(extended_timestamp)
new   1: select con_id, min(extended_timestamp), max(extended_timestamp)
old   2: from &view_prefix._AUDIT_TRAIL
new   2: from CDB_AUDIT_TRAIL
old   3: group by &con_id_col 1 order by &con_id_col 1
new   3: group by con_id, 1 order by con_id, 1

---------- ---------------------------------------- ----------------------------------------
         3 13-MAY-15 AM -06:00      13-MAY-15 PM -06:00



Finally, once we’re done testing and debugging, we can get rid of the ugly “old” and “new” statements using:


Implementing these techniques will allow modifications of most existing DBA sqlplus scripts to create universal versions, of which will be compatible with 11g (and likely earlier) databases as well as 12c legacy and container databases.


Deeper Dive

What if our monitoring query is based on an underlying catalog table and not a dictionary view?

For example, let’s say that our objective is to report on users and the last time the database password was changed. The password change date isn’t presented in the DBA_USERS or CDB_USERS view, but it is in the underlying SYS.USER$ table. Hence the monitoring query might be something like:

SQL> select name, ptime from SYS.USER$
  2  where type#=1 order by name;

NAME                     PTIME
------------------------ ---------
ANONYMOUS                23-APR-15
SYSTEM                   23-APR-15
XDB                      23-APR-15
XS$NULL                  23-APR-15


If we look at the view definition of any of the CDB_ views it is apparent that the view traverses the open PDBs by using the new 12c “CONTAINERS” function which accepts a table name as the only argument.

When run from the root container the CONTAINERS() function will traverse all open PDBs (assuming the common user used has local PDB permission to access the referenced table).

NOTE: Prior to the CONTAINERS function was called CDB$VIEW.

Thus, we can use the new function as follows:

SQL> select con_id, name, ptime from CONTAINERS(SYS.USER$)
  2  where type#=1 order by con_id, name;

    CON_ID NAME                     PTIME
---------- ------------------------ ---------
         1 ANONYMOUS                23-APR-15
         1 SYSTEM                   23-APR-15
         1 XDB                      23-APR-15
         1 XS$NULL                  23-APR-15
         3 ANONYMOUS                23-APR-15
         3 SYSTEM                   23-APR-15
         3 XDB                      23-APR-15
         3 XS$NULL                  23-APR-15


Or to make the script universal so the single script can be run on both 11g and 12c:

SQL> COLUMN view_prefix NEW_VALUE view_prefix NOPRINT
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','CONTAINERS(SYS.USER$)','SYS.USER$') view_prefix FROM v$instance;

SQL> COLUMN con_id_col NEW_VALUE con_id_col NOPRINT
SQL> SELECT DECODE(SUBSTR(version,1,INSTR(version,'.')-1),'12','con_id,','') con_id_col FROM v$instance;

SQL> select &con_id_col name, ptime from &view_prefix.
  2  where type#=1 order by &con_id_col name;
old   1: select &con_id_col name, ptime from &view_prefix.
new   1: select con_id, name, ptime from CONTAINERS(SYS.USER$)
old   2: where type#=1 order by &con_id_col name
new   2: where type#=1 order by con_id, name

    CON_ID NAME                     PTIME
---------- ------------------------ ---------
         1 ANONYMOUS                23-APR-15
         1 XDB                      23-APR-15
         1 XS$NULL                  23-APR-15
         3 ANONYMOUS                23-APR-15
         3 XDB                      23-APR-15
         3 XS$NULL                  23-APR-15



A final question might be: why isn’t the PDB$SEED database shown in the results?

The answer is that a new 12c initialization parameter EXCLUDE_SEED_CDB_VIEW controls whether the seed database is displayed in CDB_ view (or CONTAINERS() function calls). EXCLUDE_SEED_CDB_VIEW is dynamic and session modifiable:


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
exclude_seed_cdb_view                boolean     TRUE

SQL> select con_id, count(1) from cdb_users group by con_id;

    CON_ID   COUNT(1)
---------- ----------
         1         18
         3         20


Session altered.

SQL> select con_id, count(1) from cdb_users group by con_id;

    CON_ID   COUNT(1)
---------- ----------
         1         18
         2         17
         3         20



Other tools

A final question is whether this technique will still work if the SQL script is run through other tools? The answer is: “it depends“.

It depends on whether the other tools support the “define” command and the use of script variables. Specifically, Oracle SQL Developer and the newer sqlcl tool does. The above examples work fine in SQL Developer and sqlcl using the standard sqlcl “default” sqlformat. Other sqlformat options in sqlcl show some issues (testing with sqlcl version


Learn more about Pythian’s expertise in Oracle and MySQL.

Categories: Alex @ Pythian Tags:

Log Buffer #424: A Carnival of the Vanities for DBAs

May 26th, 2015 Alex Gorbachev No comments

This Log Buffer Edition covers various valuable blog posts from the fields of Oracle, SQL Server and MySQL.


  • Oracle Big Data Appliance X5-2 with Big Data SQL for the DBA.
  • Loading, Updating and Deleting From HBase Tables using HiveQL and Python.
  • In keeping with the ODA quarterly patching strategy, Appliance Manager is now available.
  • From time to time someone publishes a query on the OTN database forum and asks how to make it go faster, and you look at it and think, “it’s a nice example to explain a couple of principles because it’s short, easy to understand, obvious what sort of things might be wrong, and easy to fix.”
  • Optimizing the PL/SQL Challenge IV: More OR Condition Woes.

SQL Server:

  • Will RDBMs be obsolete? Should Data Professionals care about Big Data technologies? What is NoSQL? What is Hadoop?
  • In a development team, there are times when the relationships between developers and testers can become strained. How can you turn this potential conflict into something more positive?
  • Michael Fal is a huge advocate of automation and many ways it can improve the lives of developers and DBAs alike, but you can’t just automate all your problems away.
  • One way to handle a very complex database project with several databases and cross references.
  • Building the Ideal VMware-based SQL Server Virtual Machine.


  • Optimizing Out-of-order Parallel Replication with MariaDB 10.0.
  • General-purpose MySQL applications should read MySQL option files like /etc/my.cnf, ~/.my.cnf, … and ~/.mylogin.cnf. But ~/.mylogin.cnf is encrypted.
  • Creating and Restoring Database Backups With mysqldump and MySQL Enterprise Backup.
  • If you don’t know much about bash shell, you should start with the prior post to learn about bash arrays.
  • Installing Kubernetes Cluster with 3 minions on CentOS 7 to manage pods and services.

Learn more about Pythian’s expertise in Oracle , SQL Server and MySQL.

Categories: Alex @ Pythian Tags:

EM12c : Login to GUI with the correct password causes authentication failure

May 21st, 2015 Alex Gorbachev No comments

So the other day I was trying to log in to my EM12c R4 environment with the SSA_ADMINISTRATOR user, and I got the error:

“Authentication failed. If problem persists, contact your system administrator”

I was quite sure that the password that I had was correct, so I tried with the SYSMAN user and had the same error. I still wanted to verify that I had the correct password , so I tried with the SYSMAN user to log in to the repository database, and was successful, so I know something was wrong there.

SQL&gt; connect sysman/
Enter password:

So I went to the<gc_inst>/em/EMGC_OMS1/sysman/log/emoms.log and saw the following error

2015-05-18 21:22:06,103 [[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR audit.AuditManager auditLog.368 - Could not Log audit data, Error:java.sql.SQLException: ORA-14400: inserted partition key does not map to any partition
ORA-06512: at &quot;SYSMAN.MGMT_AUDIT&quot;, line 492
ORA-06512: at &quot;SYSMAN.MGMT_AUDIT&quot;, line 406
ORA-06512: at line 1

Which led me to believe that the JOB_QUEUE_PROCESSES was set to 0, but it wasn’t the case, since it was set to 50. Though, this is actually an incorrect limit, so I bumped it up to 1000 and tried to rerun the EM12c repository DBMS Scheduler jobs as per the documentation in 1498456.1:

SQL&gt; show parameter JOB_QUEUE_PROCESSES

------------------------------------ ----------- ------------------------------
job_queue_processes integer 50
SQL&gt; alter system set JOB_QUEUE_PROCESSES=1000 scope = both;

System altered.

SQL&gt; show parameter both
SQL&gt; show parameter job

------------------------------------ ----------- ------------------------------
job_queue_processes integer 1000
SQL&gt; connect / as sysdba
SQL&gt; alter system set job_queue_processes = 0;

System altered.

SQL&gt; connect sysman/alyarog1605
SQL&gt; exec emd_maintenance.remove_em_dbms_jobs;

PL/SQL procedure successfully completed.

SQL&gt; exec gc_interval_partition_mgr.partition_maintenance;

PL/SQL procedure successfully completed.

SQL&gt; @$OMS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_recompile_invalid.sql SYSMAN
old 11: AND owner = upper('&amp;RECOMPILE_REPOS_USER')
new 11: AND owner = upper('SYSMAN')
old 26: dbms_utility.compile_schema(upper('&amp;RECOMPILE_REPOS_USER'),FALSE);
new 26: dbms_utility.compile_schema(upper('SYSMAN'),FALSE);
old 41: WHERE owner = upper('&amp;RECOMPILE_REPOS_USER')
new 41: WHERE owner = upper('SYSMAN')
old 84: AND owner = upper('&amp;RECOMPILE_REPOS_USER')
new 84: AND owner = upper('SYSMAN')
old 104: AND ds.table_owner = upper('&amp;RECOMPILE_REPOS_USER')
new 104: AND ds.table_owner = upper('SYSMAN')

PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.

SQL&gt; connect / as sysdba
SQL&gt; alter system set job_queue_processes = 1000;

System altered.

SQL&gt; connect sysman/
Enter password:
SQL&gt; exec emd_maintenance.submit_em_dbms_jobs;

PL/SQL procedure successfully completed.

SQL&gt; commit;

Commit complete.

After this I bounced the OMS, but still kept getting the same error. And though it fixed the scheduler jobs, I was now seeing the following error in the emoms.log:

2015-05-18 22:29:09,573 [[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN auth.EMRepLoginFilter doFilter.450 - InvalidEMUserException caught in EMRepLoginFilter: Failed to login using repository authentication for user: SSA_ADMIN
oracle.sysman.emSDK.sec.auth.InvalidEMUserException: Failed to login using repository authentication for user: SSA_ADMIN

So what I did was an update to the SYSMAN.MGMT_AUDIT_MASTER table and ran the procedure MGMT_AUDIT_ADMIN.ADD_AUDIT_PARTITION as was stated in document id 1493151.1:

oracle $ sqlplus


Enter user-name: sysman
Enter password:

SQL&gt; update mgmt_audit_master set prepopulate_days=5 where prepopulate_days is null;

1 rows updated.

SQL&gt; select count(1) from mgmt_audit_master where prepopulate_days is null;


SQL&gt; exec mgmt_audit_admin.add_audit_partition;

PL/SQL procedure successfully completed.

SQL&gt; commit;

Commit complete.

Once I did this, I was able to login with all my EM12c administrators without any issues:

oracle@em12cr4.localdomain [emrep] /home/oracle
oracle $ emcli login -username=ssa_admin
Enter password

Login successful


Even though the JOB_QUEUE_PROCESSES were not set to 0, it was the cause that it was failing, as it was a low value for this parameter. Thus, be careful when setting up this parameter, be sure to follow the latest installation guidelines.

Note– This was originally published on

Categories: Alex @ Pythian Tags:

Log Buffer #423: A Carnival of the Vanities for DBAs

May 20th, 2015 Alex Gorbachev No comments

This Log Buffer edition covers Oracle, SQL Server and MySQL blog posts from all over the blogosphere!


Hey DBAs:  You know you can  install and run Oracle Database 12c on different platforms, but if you install it on an Oracle Solaris 11 zone, you can take additional advantages.

Here is a video with Oracle VP of Global Hardware Systems Harish Venkat talking with Aaron De Los Reyes, Deputy Director at Cognizant about his company’s explosive growth & how they managed business functions, applications, and supporting infrastructure for success.

Oracle Unified Directory is an all-in-one directory solution with storage, proxy, synchronization and virtualization capabilities. While unifying the approach, it provides all the services required for high-performance enterprise and carrier-grade environments. Oracle Unified Directory ensures scalability to billions of entries. It is designed for ease of installation, elastic deployments, enterprise manageability, and effective monitoring.

Understanding Flash: Summary – NAND Flash Is A Royal Pain In The …

Extracting Oracle data & Generating JSON data file using ROracle.

SQL Server:

It is no good doing some or most of the aspects of SQL Server security right. You have to get them all right, because any effective penetration of your security is likely to spell disaster. If you fail in any of the ways that Robert Sheldon lists and describes, then you can’t assume that your data is secure, and things are likely to go horribly wrong.

How does a column store index compare to a (traditional )row store index with regards to performance?

Learn how to use the TOP clause in conjunction with the UPDATE, INSERT and DELETE statements.

Did you know that scalar-valued, user-defined functions can be used in DEFAULT/CHECK CONSTRAINTs and computed columns?

Tim Smith blogs as how to measure a behavioral streak with SQL Server, an important skill for determining ROI and extrapolating trends.

Pilip Horan lets us know as How to run SSIS Project as a SQL Job.


Encryption is important component of secure environments. While being intangible, property security doesn’t get enough attention when it comes to describing various systems. “Encryption support” is often the most of details what you can get asking how secure the system is. Other important details are often omitted, but the devil in details as we know. In this post I will describe how we secure backup copies in TwinDB.

The fsfreeze command, is used to suspend and resume access to a file system. This allows consistent snapshots to be taken of the filesystem. fsfreeze supports Ext3/4, ReiserFS, JFS and XFS.

Shinguz: Controlling worldwide manufacturing plants with MySQL.

MySQL 5.7.7 was recently released (it is the latest MySQL 5.7, and is the first “RC” or “Release Candidate” release of 5.7), and is available for download

Upgrading Directly From MySQL 5.0 to 5.6 With mysqldump.

One of the cool new features in 5.7 Release Candidate is Multi Source Replication.


Learn more about Pythian’s expertise in Oracle , SQL Server and MySQL.

Categories: Alex @ Pythian Tags:

fsfreeze in Linux

May 14th, 2015 Alex Gorbachev No comments

The fsfreeze command, is used to suspend and resume access to a file system. This allows consistent snapshots to be taken of the filesystem. fsfreeze supports Ext3/4, ReiserFS, JFS and XFS.

A filesystem can be frozen using following command:

# /sbin/fsfreeze -f /data

Now if you are writing to this filesystem, the process/command will be stuck. For example, following command will be stuck in D (UNINTERUPTEBLE_SLEEP) state:

# echo “testing” > /data/file

Only after the filesystem is unfreezed using the following command, can it continue:

# /sbin/fsfreeze -u /data

As per the fsfreeze main page, “fsfreeze is unnecessary for device-mapper devices. The device-mapper (and LVM) automatically freezes filesystem on the device when a snapshot creation is requested.”

fsfreeze is provided by the util-linux package in RHEL systems. Along with userspace support, fsfreeze also requires kernel support.

For example, in the following case, fsfreeze was used in the ext4 filesystem of an AWS CentOS node:

# fsfreeze -f /mysql
fsfreeze: /mysql: freeze failed: Operation not supported

From strace we found that ioctl is returning EOPNOTSUPP:

fstat(3, {st_dev=makedev(253, 0), st_ino=2, st_mode=S_IFDIR|0755,
st_nlink=4, st_uid=3076, st_gid=1119, st_blksize=4096, st_blocks=8,
st_size=4096, st_atime=2014/05/20-10:58:56,
st_mtime=2014/11/17-01:39:36, st_ctime=2014/11/17-01:39:36}) = 0
ioctl(3, 0xc0045877, 0) = -1 EOPNOTSUPP (Operation not

From latest upstream kernel source:

static int ioctl_fsfreeze(struct file *filp)
struct super_block *sb = file_inode(filp)->i_sb;if (!capable(CAP_SYS_ADMIN))
return -EPERM;

/* If filesystem doesn’t support freeze feature, return. */
if (sb->s_op->freeze_fs == NULL)

/* Freeze */
return freeze_super(sb);

EOPNOTSUPP is returned when a filesystem does not support the feature.

On testing to freeze ext4 in CentOs with AWS community AMI, fsfreeze worked fine.

This means that the issue was specific to the kernel of the system. It was found that AMI used to build the system was having a customized kernel without fsfreeze support.

Categories: Alex @ Pythian Tags:

Ingest a Single Table from Microsoft SQL Server Data into Hadoop

May 13th, 2015 Alex Gorbachev No comments


This blog describes the best-practice approach in regards to the data ingestion from SQL Server into Hadoop. The case scenario is described as under:

  • Single table ingestion (no joins)
  • No partitioning
  • Complete data ingestion (trash old and replace new)
  • Data stored in Parquet format


This example has been tested using the following versions:

  • Hadoop 2.5.0-cdh5.3.0
  • Hive 0.13.1-cdh5.3.0
  • Sqoop 1.4.5-cdh5.3.0
  • Oozie client build version: 4.0.0-cdh5.3.0

Process Flow Diagram



  • Create the following directory/file structure (one per data ingestion process). For a new ingestion program please adjust the directory/file names as per requirements. Make sure to replace the
    tag with your table name
+ hive-<table_name>
+ oozie-properties
+ oozie-<table_name>-ingest
+ lib
  • The ingestion process is invoked using an oozie workflow. The workflow invokes all steps necessary for data ingestion including pre-processing, ingestion using sqoop and post-processing.
This directory stores all files that are required by the oozie workflow engine. These files should be stored in HDFS for proper functioning of oozie
This directory stores the <table_name>.properties. This file stores the oozie variables such as database users, name node details etc. used by the oozie process at runtime.
This directory stores a file called create-schema.hql  which contains the schema definition of the HIVE tables. This file is required to be run in HIVE only once.
  • Configure files under oozie-<table_name>-ingest
1.   Download kite-data-core.jar and kite-data-mapreduce.jar files from
2.  Download sqljdbc4.jar from
3.  Configure coordinator.xml. Copy and paste the following XML.
<coordinator-app name=”<table_name>-ingest-coordinator” frequency=”${freq}” start=”${startTime}” end=”${endTime}” timezone=”UTC” xmlns=”uri:oozie:coordinator:0.2″>
<value>${coord:formatTime(coord:nominalTime(), ‘YYYY-MM-dd’)}</value>

4.  Configure workflow.xml. This workflow has three actions:

a) mv-data-to-old – Deletes old data before refreshing new
b) sqoop-ingest-<table_name> – Sqoop action to fetch table from SQL Server
c) invalidate-impala-metadata – Revalidate Impala data after each refresh
Copy and paste the following XML.
<workflow-app name=”<table_name>-ingest” xmlns=”uri:oozie:workflow:0.2″><start to=”mv-data-to-old” /><action name=”mv-data-to-old”>
<delete path=’${sqoop_directory}/<table_name>/*.parquet’ />
<delete path=’${sqoop_directory}/<table_name>/.metadata’ />
</fs><ok to=”sqoop-ingest-<table_name>”/>
<error to=”kill”/>
</action><action name=”sqoop-ingest-<table_name>”>
<sqoop xmlns=”uri:oozie:sqoop-action:0.3″>
<delete path=”${nameNode}/user/${wf:user()}/_sqoop/*” />
</sqoop><ok to=”invalidate-impala-metadata”/>
<error to=”kill”/>
</action><action name=”invalidate-impala-metadata”>
<shell xmlns=”uri:oozie:shell-action:0.1″>
<ok to=”fini”/>
<error to=”kill”/>
<kill name=”kill”>
<message>Workflow failed with error message ${wf:errorMessage(wf:lastErrorNode())}</message>
</kill><end name=”fini” /></workflow-app>

5. Configure This file will execute commands to revalidate impala metadata after each restore. Copy and paste the following data.

export PYTHON_EGG_CACHE=./myeggs
impala-shell -i <hive_server> -q “invalidate metadata <hive_db_name>.<hive_table_name>”
  • Configure files under oozie-properties. Create file with contents as under. Edit the parameters as per requirements.
# Coordinator schedulings
# Sqoop settings
# Hive/Impala Settings
# MS SQL Server settings
  • Configure files under hive-<table_name>. Create a new file create-schema.hql with contents as under.
LOCATION ‘hdfs:///data/sqoop/<table_name>';


  • Create new directory in HDFS and copy files
$ hadoop fs -mkdir /user/<user>/oozie/<table_name>-ingest
$ hadoop fs -copyFromLocal <directory>/<table_name>/oozie-<table_name>-ingest/lib /user/<user>/oozie/ <table_name>-ingest
$ hadoop fs -copyFromLocal <directory>/<table_name>/oozie-<table_name>-ingest/ coordinator.xml /user/<user>/oozie/ <table_name>-ingest
$ hadoop fs -copyFromLocal <directory>/<table_name>/oozie-<table_name>-ingest/ /user/<user>/oozie/<table_name>-ingest
$ hadoop fs -copyFromLocal <directory>/<table_name>/oozie-<table_name>-ingest/ workflow.xml /user/<user>/oozie/ <table_name>-ingest
  • Create new directory in HDFS for storing data files
$ hadoop fs -mkdir /user/SA.HadoopPipeline/oozie/<table_name>-ingest
$ hadoop fs -mkdir /data/sqoop/<table_name>
  • Now we are ready to select data in HIVE. Go to URL http://<hive_server>:8888/beeswax/#query.
a. Choose existing database on left or create new.
b. Paste contents of create-schema.hql in Query window and click Execute.
c. You should now have an external table in HIVE pointing to data in hdfs://<namenode>/data/sqoop/<table_name>
  • Create Oozie job
a. Choose existing database on left or create new.
$ oozie job -run -config /home/<user>/<<directory>/<table_name>/oozie-properties/

Validation and Error Handling

  • At this point an oozie job should be created. To validate the oozie job creation open URL http://<hue_server>:8888/oozie/list_oozie_coordinators. Expected output as under. In case of error please review the logs for recent runs.
  • To validate the oozie job is running open URL http://<hue_server>:8888/oozie/list_oozie_workflows/ . Expected output as under. In case of error please review the logs for recent runs.
  • To validate data in HDFS execute the following command. You should see a file with *.metadata extension and a number of files with *.parquet extension.
$ hadoop fs -ls /data/sqoop/<table_name>/
  • Now we are ready to select data in HIVE or Impala.
    For HIVE go to URL http://<hue_server>:8888/beeswax/#query
    For Impala go to URL http://<hue_server>:8888/impala
    Choose the newly created database on left and execute the following SQL – select * from <hive_table_name> limit 10
    You should see the the data being outputted from the newly ingested data.
Categories: Alex @ Pythian Tags:

Log Buffer #422: A Carnival of the Vanities for DBAs

May 8th, 2015 Alex Gorbachev No comments

This Log Buffer Edition picks, choose and glean some of the top notch blog posts from Oracle, SQL Server and MySQL.


  • The standard images that come with devstack are very basic
  • Oracle is pleased to announce the release of Oracle VM VirtualBox 5.0 BETA 3
  • Monitoring Parallel Execution using Real-Time SQL Monitoring in Oracle Database 12c
  • Accessing your Cloud Integration API end point from Javascript
  • Are You Ready for The Future of Oracle Database?

SQL Server:

  • SQL Monitor Custom Metric: Plan Cache; Cache Pages Total
  • Generating A Password in SQL Server with T-SQL from Random Characters
  • This article explains how default trace can be used for auditing purposes when combined with PowerShell scripts
  • How to FTP a Dynamically Named Flat File
  • Alan Cooper helped to debug the most widely-used PC language of the late seventies and early eighties, BASIC-E, and, with Keith Parsons, developed C-BASIC. He then went on to create Tripod, which morphed eventually into Visual Basic in 1991.


  • There’s a new kid on the block in the NoSQL world – Azure DocumentDB
  • Spring Cleaning in the GIS Namespace
  • MySQL replication is among the top features of MySQL. In replication data is replicated from one MySQL Server (also knows as Master) to another MySQL Server (also known as Slave). MySQL Binlog Events is a set of libraries which work on top of replication and open directions for myriad of use cases like extracting data from binary log files, building applications to support heterogeneous replication, filtering events from binary log files and much more.
  • New to pstop – vmstat style stdout interface
  • The Perfect Server – Ubuntu 15.04 (Vivid Vervet) with Apache, PHP, MySQL, PureFTPD, BIND, Postfix, Dovecot and ISPConfig 3
Categories: Alex @ Pythian Tags:

OpenTSDB and Google Cloud Bigtable

May 6th, 2015 Alex Gorbachev No comments

Data comes in different shapes. One of the these shapes is called a time series. Time series is basically a sequence of data points recorded over time. If, for example, you measure the height of the tide every hour for 24 hours, then you will end up with a time series of 24 data points. Each data point will consist of tide height in meters and the hour it was recorded at.

Time series are very powerful data abstractions. There are a lot of processes around us that can be described by a simple measurement and a point in time this measurement was taken at. You can discover patterns in your website users behavior by measuring the number of unique visitors every couple of minutes. This time series will help you discover trends that depend on the time of day, day of the week, seasonal trends, etc. Monitoring a server’s health by recording metrics like CPU utilization, memory usage and active transactions in a database at a frequent interval is an approach that all DBAs and sysadmins are very familiar with. The real power of time series is in providing a simple mechanism for different types of aggregations and analytics. It is easy to find, for example, minimum and maximum values over a given period of time, or calculate average, sums and other statistics.

Building a scalable and reliable database for time series data has been a goal of companies and engineers out there for quite some time. With ever increasing volumes of both human and machine generated data the need for such systems is becoming more and more apparent.

OpenTSDB and HBase

There are different database systems that support time series data. Some of them (like Oracle) provide functionality to work with time series that is built on top of their existing relational storage. There are also some specialized solutions like InfluxDB.

OpenTSDB is somewhere in between these two approaches: it relies on HBase to provide scalable and reliable storage, but implements it’s own logic layer for storing and retrieving data on top of it.

OpenTSDB consists of a tsd process that handles all read/write requests to HBase and several protocols to interact with tsd. OpenTSDB can accept requests over Telnet or HTTP APIs, or you can use existing tools like tcollector to publish metrics to OpenTSDB.

OpenTSDB relies on scalability and performance properties of HBase to be able to handle high volumes of incoming metrics. Some of the largest OpenTSDB/HBase installations span over dozens of servers and process ~280k writes per second (numbers from

There exist a lot of different tools that complete OpenTSDB ecosystem from various metrics collectors to GUIs. This makes OpenTSDB one of the most popular ways to handle large volumes of time series information and one of the major HBase use cases as well. The main challenge with this configuration is that you will need to host your own (potentially very large) HBase cluster and deal with all related issues from hardware procurement to resource management, dealing with Java garbage collection, etc.

OpenTSDB and Google Cloud Bigtable

If you trace HBase ancestry you will soon find out that it all started when Google published a paper on a scalable data storage called Bigtable. Google has been using Bigtable internally for more than a decade as a back end for web index, Google Earth and other projects. The publication of the paper initiated creation of Apache HBase and Apache Cassandra, both very successful open source projects.

Latest release of Bigtable as a publicly available Google Cloud service gives you instant access to all the engineering effort that was put into Bigtable at Google over the years. Essentially, you are getting a flexible, robust HBase-like database that lacks some of the inherited HBase issues, like Java GC stalls. And it’s completely managed, meaning you don’t have to worry about provisioning hardware, handling failures, software installs, etc.

What does it mean for OpenTSDB and time series databases in general? Well, since HBase is built on Bigtable foundation it is actually API compatible with Google Cloud Bigtable. This means that your applications that work with HBase could be switched to work with Bigtable with minimal effort. Be aware of some of the existing limitations though. Pythian engineers are working on integrating OpenTSDB to work with Google Cloud Bigtable instead of HBase and we hope to be able to share results with the community shortly. Having Bigtable as a back end for OpenTSDB opens a lot of opportunities. It will provide you with a managed cloud-based time-series database, which can be scaled on demand and doesn’t require much maintenance effort.

There are some challenges that we have to deal with, especially around a client that OpenTSDB uses to connect to HBase. OpenTSDB uses it’s own implementation of HBase client called AsyncHBase. It is compatible on a wire protocol level with HBase 0.98, but uses a custom async Java library to allow for asynchronous interaction with HBase. This custom implementation allows OpenTSDB to perform HBase operations much faster than using standard HBase client.

While HBase API 1.0.0 introduced some asynchronous behavior using BufferedMutator it is not a trivial task to replace AsyncHBase with a standard HBase client, because it is tightly coupled with the rest of OpenTSDB code. Pythian engineers are working on trying out several ideas on how to make the transition to standard client look seamless from an OpenTSDB perspective. Once we have a standard HBase client working, connecting OpenTSDB to Bigtable should be simple.

Stay tuned.

Categories: Alex @ Pythian Tags:

OEM 12c Silent Installation

May 4th, 2015 Alex Gorbachev No comments

“What’s for lunch today?”, said the newly born ready to run Red Hat 6.4 server.

“Well, I have an outstanding 3-course meal of OEM12c installation.
For the appetizer, a light and crispy ASM 12c,
DB 12c with patching for the main and desert, and to cover everything up, OEM 12c setup and configuration”, replied  the DBA who was really happy to prepare such a great meal for his new friend.

“Ok, let’s start cooking, it won’t take long”, said the DBA and took all his cookware (software), prepared ingredients (disk devices) and got the grid infrastructure cooked:

./runInstaller -silent \
-responseFile /home/oracle/install/grid/response/grid_install.rsp -showProgress \
INVENTORY_LOCATION=/u01/app/oraInventory \
oracle.install.option=HA_CONFIG \
ORACLE_BASE=/u01/app/oracle \
ORACLE_HOME=/u01/app/oracle/product/12.1.0/grid \
oracle.install.asm.OSDBA=dba \
oracle.install.asm.OSASM=dba \ \
oracle.install.asm.SYSASMPassword=sys_pwd \ \
oracle.install.asm.diskGroup.redundancy=EXTERNAL \
oracle.install.asm.diskGroup.AUSize=4 \
oracle.install.asm.diskGroup.disks=/dev/asm-disk1 \
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/asm* \
oracle.install.asm.monitorPassword=sys_pwd \

And added some crumbs:

/u01/app/oracle/product/12.1.0/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/tmp/asm.rsp
where /tmp/asm.rsp had:

“It was a great starter”, said the server finishing the first dish,

“I am getting even more hungry. What’s for the main?”.

“Oh, you will love it! It is Database 12c. It is one of these new meals and it is already very popular”, answered the DBA enthusiastically and continued cooking.

“Looking forward to trying it”, the server decided to have a nap until the dish was ready.

“You asked, you got it”, and the DBA gave the server the dish he never tried:

./runInstaller -silent -showProgress \
-responseFile /home/oracle/install/database/response/db_install.rsp \
oracle.install.option=INSTALL_DB_SWONLY \
ORACLE_BASE=/u01/app/oracle \
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 \
oracle.install.db.InstallEdition=EE oracle.install.db.DBA_GROUP=dba \
oracle.install.db.BACKUPDBA_GROUP=dba \
oracle.install.db.DGDBA_GROUP=dba \
oracle.install.db.KMDBA_GROUP=dba \

The topping ingredient was of course a brand new database:

./dbca -silent -createDatabase -gdbName em12 \
-templateName General_Purpose.dbc \
-emConfiguration none \
-sysPassword sys_pwd \
-systemPassword sys_pwd \
-storageType ASM \
-asmsnmpPassword sys_pwd \
-diskGroupName DATA \
-redoLogFileSize 100 \
-initParams log_buffer=10485760,processes=500,\
session_cached_cursors=300,db_securefile=PERMITTED \
-totalMemory 2048

“Delicious! That’s what I dreamt of! Where did you find it?”, the server could not hide his admiration.

“Well, you have not tried desert yet. When you have it, you will forget all those dishes that you had before.”

“Hmm, you intrigue me. Definitely I will have it!”

“Anything for you, my friend”, and the DBA cooked his famous, rich and delicious desert:

./runInstaller -silent \
-responseFile /home/oracle/install/em/response/new_install.rsp \
-staticPortsIniFile /tmp/ports.ini \
AGENT_BASE_DIR=/u01/agent12c \
SYS_PASSWORD=sys_pwd \

“You made my day!” exclaimed the server when nothing was left on his plate.

“Anytime my friend!” smiled DBA in response.

He was as happy as any chef that the cooking went the way it was planned and the final product was just as the recipe had said.

Have a good day!

Categories: Alex @ Pythian Tags:
  • video recorder free download for android
  • android app to download free books
  • can android phones download whatsapp
  • download leitor qr code para android
  • download king james bible for android
  • nova hd for android free download
  • my paper plane 2 download android
  • android nand rom hd2 download
  • download captain america for android apk
  • android games download via pc
  • better android better keyboard app download
  • 6227760957
  • 3d google maps android download
  • best youtube downloader on android
  • raging thunder 2 full free download android
  • 7933757448
  • scarica youtube downloader per android
  • games android download gratis
  • google voice android download free
  • download drawing application for android
  • crack wpa alice android download
  • android netbook download iso
  • 4 player reactor download android free
  • 2041720016
  • 1554244475
  • nook android application download
  • invalid download url android market
  • google navigation app for android download
  • android sdk avd manager slow download
  • android download acer d255
  • rockplayer lite android app download
  • free country music downloads for android
  • download free french dictionary for android
  • android market app download xda
  • android apps download freeware
  • best mp3 music download app for android
  • adventure of tintin android game download
  • 4892610380
  • download aldiko book reader for android free
  • fifa 10 download free for android
  • red alert 2 for android free download
  • dictionar roman englez pentru android download
  • android apps download barcode scanner
  • download game bejeweled 2 cho android
  • 7794973540
  • forest runner apk android download
  • android app development file download
  • mp3 download app for android free
  • download usb driver for android samsung
  • go locker android download free
  • spb shell 3d android download apk
  • fifa 10 for android download free
  • iplayer download to android
  • android launcher theme download
  • flv downloader for android
  • virtua tennis android game download
  • tap the frog android download free
  • android market error download was unsuccessful
  • download movies to your android phone
  • pes 2011 android full download free
  • samsung omnia sgh i900 android download
  • kick the boss 2 hack android download
  • picasa android app download
  • applicazioni android per scaricare film
  • best things to download on android
  • night vision cam download android
  • aldiko book reader android app
  • talking tom cat 1 free download android
  • download lagu free untuk android
  • jelly defense android download
  • free ebooks to download for android
  • talking tomcat android app download
  • android hsusb device no driver found
  • android touchpad alpha 3 download
  • is mp3 download on android legal
  • android lock xt download for ipod
  • 6724637911
  • 8187683779
  • mobile voip software free download for android
  • blow up full android download
  • bubble level app android download
  • fifa street android apk download
  • download video from browser android
  • alchemy android download game pc
  • download android untuk notebook
  • applications to download videos on android
  • avi player for android mobile free download
  • where downloads go on android
  • 8010560689
  • chrome android default download location
  • little photo app android free download
  • skype for android download file
  • android market free download for lg
  • das inoffizielle android handbuch free download
  • android for lg download
  • most downloaded apps for android 2010
  • the curse of monkey island download android
  • facebook zynga poker android download
  • star trek ringtones android download
  • good app download music android
  • tube music android app download
  • alcatel 918d android manager download
  • how to download infinity blade on android
  • android app download to computer
  • download do sistema android para netbook
  • download cepat untuk android
  • slam dunk king android download
  • gta game download for android
  • alchemy android download pc
  • download videos on youtube android
  • android themes for nokia c5 00
  • antivirus for android mobile download
  • android 4 download for xperia
  • google music android download to device
  • mp3 youtube downloader android
  • leitor de qr code para android
  • mixzing media player android download
  • download smosh shut up app android
  • bb messenger for android free download
  • cooking games for android
  • 2576963315
  • android apps download free apk
  • call cheater apk for android free download
  • android will not download forecasting
  • trial xtreme android game download
  • der bro code android download
  • spy software free download for android
  • copilot android canada map download
  • clock for android free download
  • ebook reader android apk download
  • pro android media download
  • just dance android app download
  • android x plane 9 download
  • chuong trinh download video cho android
  • c cleaner for android free download
  • flash development for android cookbook penns download
  • como baixar coisas no android
  • pokemon tower defence download android
  • document viewer android free download
  • 4357669836
  • bat link download tren android
  • freedi youtube downloader apps android
  • download roms android psx
  • can u download itunes on android
  • android operating system free download for tablet
  • cara download video dari youtube pada android
  • apps you should download for android
  • sony ericsson pc suite download for android
  • unable download apps android market
  • android free music download legal
  • android os free download for nokia
  • nba 2k12 for android apk free download
  • mid android tablet apps free download
  • adb driver for android windows xp
  • dungeon hunter android download apk
  • android adb command line tool download
  • raging thunder 2 android game free download
  • tablet android software download
  • akinator download android gratis
  • 8104214547
  • winamp pro android download gratis
  • programa download de musicas para android
  • pocket informant android apk download
  • android download sdk source
  • dropbox android download a folder
  • free download avatar hd game for android
  • android developer download file internet
  • age of zombies android download chomikuj
  • amazon mp3 android download all
  • best free downloads for android
  • android 4 download for htc sensation
  • htc elf android rom download
  • android won't download large files
  • digital clock widget android download apk
  • offline maps download android
  • book of mormon download for android
  • download nes games for android phone
  • download google body browser android
  • android market aktualizacja download
  • google maps for android download
  • android ums composite usb device driver download
  • touch pro 2 android download
  • youtube downloader in android market
  • download game animal kaiser for android
  • android app flash player download
  • download android apps via computer
  • aldiko book reader for android free download
  • android temporary download folder
  • amazon kindle free download for android
  • no lock android free download
  • download facebook places app for android
  • bwin app download android
  • android apps music download
  • android market application download free for mobile
  • app fA?r downloadvolumen android
  • 8753014555
  • movie maker software download for android
  • aplikacje na android za darmo pobierz
  • download android market to htc hd2
  • abn amro android app downloaden
  • arc media player for android free download
  • most downloaded app of all time android
  • android app to download library books
  • aplikasi download youtube lewat hp android
  • youtube downloader fA?r android kostenlos
  • android download offline maps from google
  • 6763140452
  • qq messenger download for android mobile
  • android apk download freeware
  • minecraft pocket edition download free android apk
  • firefox browser for android download
  • free download for android games
  • 9275551636
  • can't download instagram on android
  • free download gps for android software
  • download image from url android
  • infinity blade 2 download for android
  • facebook for android new version free download
  • hp 50g para android download
  • youtube download for android apk
  • pes 2012 for android free download apk
  • image download from url in android
  • 4810006094
  • download nada dering untuk android
  • android download sd card default
  • sonic 4 episode 1 android download free
  • free video songs download for android mobile
  • mario free download for android phone
  • contract killer android game free download
  • 7660063968
  • remember the milk download android
  • 2go latest version download for android
  • osmos hd apk free download android
  • android mtp driver download galaxy tab
  • free download games balap mobil for android
  • download mx player for android with codec
  • android ics for htc hd2 download
  • lg android download unsuccessful
  • android overclock apk free download
  • error download on android
  • download epub reader for android tablet
  • phonegap download file android
  • battlefield bc 2 android free download
  • doodle jump hack download android
  • 2268018547
  • papago m8 android crack download
  • android download image from server sample
  • cisco jabber video android download
  • best free android download manager
  • descargar odin multi downloader android gratis
  • download aggiornamento android per htc desire hd
  • window 8 download for android
  • vlc player free download for android mobile
  • gps status & toolbox android download
  • player pro download android free
  • download toi app for android
  • best android games apk file download
  • download angry birds android market
  • market enabler free download for android
  • download avg antivirus pro android
  • nfs carbon game for android free download
  • aktualizacja androida lg gt540 pobierz
  • android google maps download offline
  • gry na androida za darmo pobierz
  • taskbar in android apk download
  • akinator genie android download
  • android download image from server and show
  • doubletwist player android apk download
  • youtube mp4 video downloader for android
  • football android games free download
  • gangstar miami vindication hd android download gratis
  • blow up full android download
  • navigon europe android free apk
  • cara download youtube di android froyo
  • download all texts from android
  • free android software download for pc
  • crusaders of destiny android download
  • gry download na android
  • xbox emulator free download for android
  • android magazine loadable free download
  • free download 100 floors game for android
  • android cab file download
  • download movie from android to mac
  • android market htc desire download unsuccessful
  • flash player android samsung galaxy download
  • virtua tennis challenge apk download for android
  • download do jogo max payne para android
  • android download mode drivers
  • download mx video player for android
  • download garmin for android apk
  • can you download podcast to android
  • best free android 3d games download
  • youtube app free download for android
  • kindle download interrupted android
  • sopcast player download android
  • gta san andreas na androida download
  • facetime app for android free download
  • air attack android download
  • download images facebook android
  • boot screen android download
  • offline android apps download
  • download android for alcatel ot 806
  • android apps download slow
  • cara download papago ke android
  • stopwatch app for android free download
  • 4144185742
  • top 10 apps for android phones
  • igo myway android europe download
  • free bible concordance download for android
  • whats up app free download for android
  • skype video android apk download
  • windows 7 android apk download
  • open home android apk download
  • panda themes for android free download
  • rope cut android download apk
  • android x86 fA?r pc download
  • portable wifi hotspot android download
  • android book download free
  • good app for downloading music on android
  • samsung android mobile software free download
  • android how to download pictures from internet
  • street fighter game download android
  • wattpad download on android
  • panda app download for android
  • free album downloads for android phones
  • sites download android apps free
  • rom download sites for android
  • download game asphalt 7 for android
  • espn for android app download
  • google earth download free 2010 for android
  • download odin android samsung s2
  • skype download for android tablet
  • android voice command app download
  • download autocad for android tablet
  • where does dropbox download to android
  • 1312969380
  • solo lite android full version download
  • action voip for android phone download
  • wo ist walter app android download
  • cut the rope experiments download android
  • mp3 songs free download for android
  • text plus android apk download
  • games to download for android phone
  • 5685294175
  • android composite usb driver download
  • android download free software for mobile
  • emulador de gba para android download
  • download jogo asphalt 5 para android gratis
  • brain age test free download android
  • spb time full download android
  • progress bar download android
  • arabic keyboard android download
  • android download yahoo messenger
  • android download apps using barcode
  • musica para android download
  • where to download android bar files
  • google earth map download for android
  • android phone pc suite free download
  • ipl game free download for android
  • arabic android apk download
  • can't download paid apps android market
  • dark knight rises android apk download
  • download dictionary english to persian for android
  • android live wallpaper starter kit download
  • 5893456795
  • download android cupcake source
  • 3d wallpapers free download for android mobile
  • free download adobe air untuk android
  • virtual dj android apk free download
  • sims 3 android tablet download
  • android os configure software free download
  • how to stop unwanted downloads on android
  • the most downloaded android apps
  • winzip for android mobile free download
  • vampire romance game download for android
  • samsung android software download for pc
  • 259216319
  • pokemon gold download android
  • android to download itunes
  • pandigital android download apps
  • house of dead free download for android
  • free mp3 full song downloads for android
  • android sdk linux ubuntu download
  • 7390056422
  • gps test plus android download
  • 9985762229
  • madden nfl 2012 for android free download
  • chrono trigger android apk download
  • adobe fasb ebook reader for android
  • tablet games free download android
  • mp3 music download app for android free
  • apk files for android tablet free download
  • android ndk download windows xp
  • android play music download location
  • download construction games for android
  • errore durante il download android market
  • android software download for phone
  • navigacija igo za android download free
  • android gta san andreas free download
  • gangstar miami vindication android download full
  • aplikasi download youtube untuk android
  • android ringtone free ebook
  • free download os android for laptop
  • 4 player reactor download android free
  • samurai 2 vengeance download android
  • aplicativo android download de musica
  • download aplikasi tv hp android
  • opera mini 7 android apk download
  • free car games download for android mobile
  • download driver para usb android
  • download hotspot shield for android 4
  • 3d wallpaper free download for android mobile
  • download gambino poker for android
  • can you put itunes on android
  • 3181162203
  • android tether pro free download
  • 3661875119
  • wallpaper changer for android download
  • doodle jump apk android
  • android app download failed
  • micromax android apps download
  • download game hp pes 2012 for android
  • android download save xml file
  • epidemic android game download
  • 9549572219
  • best android download site free
  • regina 3d launcher android download
  • download playbook android applications
  • avatar android game data download
  • download android apps onto computer
  • download four pics one word android
  • gameboy emulator android games download
  • modern combat 2 android download pandaapp
  • fifa 2011 download free android
  • android rom fA?r htc hd7 download
  • 4008906332
  • android jelly bean for pc download
  • android ebook reader free download
  • 5620671098
  • cara download games di android market
  • how to change download location on android
  • netflix android download apk
  • screen capture android app download
  • 9498077029
  • android hd games apk and data download
  • pc suite android huawei download
  • android games galaxy tab free download
  • download cartoon camera per android
  • app world free download for android
  • android para celular samsung baixar gratis
  • smart keyboard pro free download android
  • android themes apk download free
  • candy crush saga download for android tablet
  • quran android apk free download
  • download android market for lg gt540
  • samsung galaxy mini s5570 android market download
  • apn on off android download
  • android download acer d255
  • download yoo ninja para android
  • easytether pro download android
  • 4029730395
  • street fighter 4 apk android download
  • best free android music download app 2012
  • better keyboard 8 android download
  • htc flip clock download android
  • can you download vuze on android
  • apk download free android
  • android google calendar widget download
  • android application download youtube videos
  • call block unlimited android download
  • android games 2012 download free
  • android apps download samsung galaxy s3
  • google android download live cd
  • darksiders para android download
  • best android applications to download free
  • android task manager pro free download
  • download cds on android
  • download giochi per cellulare android
  • android applications free download xperia
  • red alert android game free download
  • android music download app 2012
  • play store application download for android
  • download amazon app store android tablet
  • sp flash tool android download
  • car racing games for android phones download
  • pokemon white rom download android
  • android launcher pro plus unlocker download
  • download video dari youtube pake android
  • download aplikasi facebook untuk android gratis
  • android market download package file invalid