Oracle: Sql script to check tablespace size

Every once in a while one of my test jobs fails, throwing this error that the table-space its created in has no further space to allocate for this table. And then comes the next sets of steps. Find out the other table space which is free.
Here is a very basic sql scripts which I use to list down all the tablespaces and their size in human readable format.

SELECT /* + RULE */  
df.tablespace_name AS "Tablespace",
df.bytes / (1024 * 1024 * 1024) AS "Size (GB)",
Trunc(fs.bytes / (1024 * 1024 * 1024)) AS "Free (GB)"       
FROM 
(
SELECT 
tablespace_name,
Sum(bytes) AS bytes 
FROM 
dba_free_space
GROUP BY
tablespace_name
) fs, 
( 
SELECT 
tablespace_name,
SUM(bytes) AS bytes
FROM 
dba_data_files
GROUP BY 
tablespace_name

) df
WHERE 
fs.tablespace_name = df.tablespace_name
ORDER BY 3 desc

Oracle : Delete duplicate records from a table

This is a very simple script to perform a task which almost every other database developer faces in his day to day job. To finding and deleting duplicate records. I would try to answer it in 3 simpler steps

Step1: Lets say you want to find the duplicate records for a pair of attributes c1 and c2
Find all duplicate records:


SELECT c1, c2,count(1) FROM MyTable GROUP BY c1,c2

Step 2: Finding the row that you want to keep. Lets say  you want to keep only the maximum value of some indentifier column




SELECT c1, c2,max(id) FROM MyTable GROUP BY c1,c2



Step 3: keep that and delete others
Delete from mytable
where id not in (
SELECT c1, c2,max(id)


FROM MyTable
GROUP BY c1,c2


)

Java Code: URL Shortener

Wrote this code some time back for a programming interview, sharing it for everyone's use.
Its a 64 character encoded URL shortener code written in java. The code does not use any database backend but can be easily modified to support that.

import java.util.HashMap;
import java.util.Random;

/*
* URL Shortener
*/
public class URLShortener {
//storage for generated keys
private HashMap keyMap;  //key-url map
private HashMap valueMap;//url-key map to quickly check whether an url is already entered in our system
private String domain; //Use this attribute to generate urls for a custom domain name defaults to http://fkt.in
private char myChars[]; //This array is used for character to number mapping
private Random myRand;  //Random object used to generate random integers
private int keyLength;  //the key lengh in url defaults to 8

//Default Constructor
URLShortener()
{
keyMap= new HashMap();
valueMap= new HashMap();
myRand= new Random();
keyLength=8;
myChars=new char[62];
for(int i=0;i<62;i++)
  {
   int j=0;
   if(i<10)
   {
    j=i+48;
   }
   else if(i >9 && i<=35)
{
j=i+55;
}
else
{
j=i+61;
}
myChars[i]=(char)j;
}
domain="http://fkt.in";
}


//Construtor which enables you to define tiny url key length and base url name  
URLShortener(int length, String newDomain)
{
this();
this.keyLength=length;
if(!newDomain.isEmpty())
{
newDomain=sanitizeURL(newDomain);
domain=newDomain;
}
}



//shortenURL
//the public method which can be called to shorten a given URL
public String shortenURL(String longURL)
{
String shortURL="";
if(validateURL(longURL))
{
longURL=sanitizeURL(longURL);
if(valueMap.containsKey(longURL))
{
shortURL=domain+"/"+valueMap.get(longURL);
}
else
{
shortURL = domain + "/" + getKey(longURL);
}
}
//add http part
return shortURL;
}

//expandURL
//public method which returns back the original URL given the shortened url
public String expandURL(String shortURL)
{
String longURL="";
String key=shortURL.substring(domain.length()+1);
longURL=keyMap.get(key);
return longURL;
}

//Validate URL
//not implemented, but should be implemented to check whether the given  URL
// is valid or not
boolean validateURL(String url)
{
return true;
}

//sanitizeURL
//This method should take care various issues with a valid url
//e.g. www.google.com,www.google.com/, http://www.google.com, http://www.google.com/
// all the above url should point to same shortened url
// There could be several other cases like these.
String sanitizeURL(String url)
{
if(url.substring(0, 7).equals("http://"))
url=url.substring(7);

if(url.charAt(url.length() -1)=='/')
url=url.substring(0,url.length()-1);
return url;
}
/*
* Get Key method
*/
private String getKey(String longURL)
{
String key;
key=generateKey();
keyMap.put(key,longURL);
valueMap.put(longURL,key);
return key;
}

//generateKey
private String generateKey()
{
String key="";
int counter=0;
boolean flag=true;
while(flag)
{
counter++;
key="";
for(int i=0;i<=keyLength;i++)
{
key+=myChars[myRand.nextInt(62)];
}
//System.out.println("Iteration: "+ counter + "Key: "+ key);
if(!keyMap.containsKey(key)){flag=false;}
}
return key;
}

//test the code
public static void main(String args[])
{
URLShortener u= new URLShortener(5,"www.tinyurl.com/");
String urls[]=
{  "www.google.com/",
"www.google.com",
"http://www.yahoo.com",
"www.yahoo.com/",
"www.amazon.com",
"www.amazon.com/page1.php",
"www.amazon.com/page2.php",
"www.flipkart.in",
"www.rediff.com",
"www.techmeme.com",
"www.techcrunch.com",
"www.lifehacker.com",
"www.icicibank.com"
};

for(int i=0;i
{
System.out.println("URL:" + urls[i] + "\tTiny: "+ u.shortenURL(urls[i]) + "\tExpanded: " + u.expandURL(u.shortenURL(urls[i])));
}
}

Mysql: Primer From Google Code University

Introduction to Databases and MySQL - Google Code University - Google Code

Getting Started with MySQL

The first thing you need to do to start learning MySQL is get access. We'll assume that you have access and can start up the command line interface. It will look something like:

mysql -uroot -ppassword

You should see the following:

Welcome to the MySQL monitor. Commands end with ; or \q.
Your MySQL connection id is 2129621 to server version: 3.23.58

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

In the exercises that follow, we will create a database and a set of tables for Plastronics order-processing. Then, we'll learn the basic commands for entering, editing and manipulating orders, customers, etc. We will spend most of our time learning how to query these tables, since that's the most interesting part.

If you have worked with other SQL systems or with MySQL in other environments, the following articles and documents may be helpful:

* Comparison of SQL Server and MySQL
* Comparison of Oracle SQL and MySQL
* MySQL Documentation

Setting Up the Tables

Step one is to create a database which is a container for our tables. Enter the following command:

create database plastronics;

Note that the commands and names are not case-sensitive. Also, note that the ending semi-colon is required or you'll see something like this:

mysql> create database plastronics
->

If this happens to you, just type the semi-colon on the line with the "->" and press Enter.

mysql> create database plastronics
-> ;
Query OK, 1 row affected (0.00 sec)

mysql>

The "Query OK" is your signal that the command was accepted and the task performed. Creating a database does not select it for use; you must do that explicitly. Enter the following command:

use plastronics;

The system will respond "Database changed". Now, we can work with the database. Let's take a minute to review the final structure we designed for the order processing database.

relat4.PNG

We need to set up four tables, that relate to one another as defined in the structure above. We use the CREATE TABLE command which allows us to specify the fields and their contents, as well as primary keys and constraints. Here is the command to create the customer table:

create table customer (cust_no int not null auto_increment, name varchar(40),
address varchar(40), phone varchar(12), primary key(cust_no));

After the "create table" part of the command, we name our table "customer". Then, in parentheses, we define the fields. The minimum information required is a fieldname, and a type, indicating what kind of data we want to store in the field.

For cust_no, we want an integer (whole number), which is abbreviated "int" in MySQL. Then, we indicate that cust_no cannot be null (which means empty) and we would like the system to fill the value in for us by auto-incrementing from the previous value, every time we insert a new record.

The other fields are of type "varchar" which means variable-length strings, or sequences of characters. The numbers following "varchar" in the command above indicate maximum length for the data stored in the field. So, the name, address and phone fields are all sequences of characters with maximum lengths 40, 40 and 12.

Finally, we set cust_no to be the primary key.

You can find more information on types and how to use them in the MySQL documentation. For more details on the CREATE TABLE command, check the MySQL reference.

Here is the command to create the orders tables:

create table orders (order_no int not null auto_increment, FK_cust_no int not null,
foreign key(FK_cust_no) references customer(cust_no), primary key(order_no));

There are only two fields in this table. The order_no field is a primary key, and is an integer, not null and it will auto increment. The cust_no field is a foreign key. We have named it FK_cust_no in the orders table to distinguish it from the cust_no field in the customer table.

Recall that a foreign key is a field that references a primary key in another table. In the command, we indicate that the FK_cust_no field is a foreign key referencing the cust_no field in customer, indicated by the "foreign key(FK_cust_no) references customer(cust_no)" part of the command. By setting the table up this way, MySQL will enforce constraints, that is, any order that we enter into the orders table must reference a valid customer in the customer table. If we enter a cust_no in orders that does not exist in customers, an error will result.

Note: If you want MySQL to enforce foreign key constraints, you need to add "type=InnoDB" at the end of the CREATE TABLE statement as in:

create table orders (order_no int not null auto_increment, FK_cust_no int not null,
foreign key(FK_cust_no) references customer(cust_no), primary key(order_no)) type=InnoDB;

"type=InnoDB" may or may not be available in your MySQL installation (Note: in some distribution, to avoid "ERROR 1005", you have to add "type=InnoDB" on both table creation statement, customer and orders). Support for InnoDB tables requires a specific compilation parameter when compiling MySQL from source.

Let's see what we have so far. Enter the following command:

show tables;

Then enter the following command to look at the structure of the orders table:


describe orders;

Here is the command to create the items table:

create table items (item_no int not null auto_increment, price int, primary key(item_no));

Practice: Take a couple minutes to make sure you understand all the parts of the command for creating the items table. Using the CREATE TABLE command, create the fourth table, which has two foreign key fields and the count. Call this table "item_details".

Here is the command for creating item_details:

create table item_details (FK_order_no int not null, foreign key(FK_order_no)
references orders(order_no), FK_item_no int not null, foreign key(FK_item_no)
references items(item_no), count int);

If your table does not match the structure defined in this command, just delete it by entering:

drop table item_details;

Then, enter the command given above for creating item_details.

Entering and Updating Data

Now that we have our tables created, we need to populate them with data. Enter the following commands. Just copy the whole block and paste at your MySQL command prompt.

insert into customer set name="Joe Boo", address="123 West", phone="412-773-5322";
insert into customer set name="Rich Wrack", address="332 East", phone="412-773-8374";
insert into customer set name="Ken Bend", address="225 Main", phone="412-773-9822";
insert into customer set name="Kim Slim", address="415 Bent", phone="412-773-6721";
insert into customer set name="Tom Plom", address="633 North", phone="412-773-4156";
select * from customer;

The INSERT command sets the fields to the corresponding values. The SELECT command with a "*" outputs all the data. Notice how the cust_no field auto incremented. Next, we populate the items table:

insert into items set price=666;
insert into items set price=700;
insert into items set price=450;
insert into items set price=1200;
select * from items;

Now, the orders table:

insert into orders set FK_cust_no=4;
insert into orders set FK_cust_no=3;
insert into orders set FK_cust_no=4;
insert into orders set FK_cust_no=1;
insert into orders set FK_cust_no=2;
insert into orders set FK_cust_no=1;
insert into orders set FK_cust_no=2;
insert into orders set FK_cust_no=3;
insert into orders set FK_cust_no=4;
insert into orders set FK_cust_no=5;
select * from orders;

Finally, the item_details table:

insert into item_details set FK_order_no=1, FK_item_no=4, count=12;
insert into item_details set FK_order_no=1, FK_item_no=3, count=56;
insert into item_details set FK_order_no=2, FK_item_no=1, count=10;
insert into item_details set FK_order_no=3, FK_item_no=2, count=43;
insert into item_details set FK_order_no=3, FK_item_no=4, count=16;
insert into item_details set FK_order_no=4, FK_item_no=2, count=87;
insert into item_details set FK_order_no=5, FK_item_no=1, count=62;
insert into item_details set FK_order_no=5, FK_item_no=2, count=48;
insert into item_details set FK_order_no=5, FK_item_no=3, count=5;
insert into item_details set FK_order_no=6, FK_item_no=3, count=87;
insert into item_details set FK_order_no=7, FK_item_no=2, count=32;
insert into item_details set FK_order_no=7, FK_item_no=1, count=27;
insert into item_details set FK_order_no=8, FK_item_no=4, count=91;
insert into item_details set FK_order_no=9, FK_item_no=2, count=34;
insert into item_details set FK_order_no=9, FK_item_no=3, count=72;
insert into item_details set FK_order_no=10, FK_item_no=4, count=2;
select * from item_details;

If you need to edit a record, MySQL provides an UPDATE command:

update item_details set count=12 where FK_order_no=7 and FK_item_no=2;
select * from item_details where FK_order_no=7 and FK_item_no=2;

Notice how we can define the exact record for both UPDATE and SELECT using the WHERE clause. We have also used the AND connector. UPDATE can also be used to edit a group of records. For example, we could set count=12 for both of the items in order #7:

update item_details set count=12 where FK_order_no=7;
select * from item_details where FK_order_no=7;

We can also DELETE in a similar manner:

delete from item_details where FK_order_no=7;
select * from item_details where FK_order_no=7;

Let's put those records back in now:

insert into item_details set FK_order_no=7, FK_item_no=2, count=32;
insert into item_details set FK_order_no=7, FK_item_no=1, count=27;

For more information on these MySQL commands, check the MySQL Documentation.

Practice: See if MySQL checks for primary key constraints by trying to insert a new record in a table with a primary key that has the same primary key value as a record already in the table. Note that foreign key constraints are only checked if you created the tables using "type=InnoDB".

Doing Queries

We have already seen the use of the SELECT command. This is the command we use to query tables. We can see all the fields in a table using "*" or we can specify certain fields. We can also limit the output using the WHERE clause. Here are some examples:

mysql6.PNG

Practice: See if you can create SELECT statements to find the following data.

* All orders that include item #4 (only list the order numbers once). Answer: orders #1, 3, 8, 10.
* All orders for customer #2. Answer: orders #5 and 7
* Try entering a query that returns no results. How does MySQL communicate this?

We can also search within varchar fields using LIKE and the "%" wildcard:

select * from customer where address like "%West";
select * from customer where name like "%o%";

This last command outputs all records where there is an "o" anywhere in the name field. Note that using LIKE is very slow on large databases - this is not something you would do on the Ads database.

The real power of the SELECT command becomes evident when we join tables using foreign key relationships. In order to illustrate how this works, we need to add some records to the orders table:

insert into orders set FK_cust_no=10;
insert into orders set FK_cust_no=11;

Notice that we have entered orders for customers that do not exist in the customer table.

Take a look at the following command:

mysql7.PNG

In this SELECT, we have combined the orders and customer tables using the customer number. This is called an inner join. We are listing fields from both tables, and the ON clause makes the connection between the primary and foreign key. Notice that the records are listed in customer number order. Thus, an inner join takes all the records from the orders table and finds the matching record(s) in the customer table.

Another type of join is the left join. It returns all the values from left table (orders) and the matched values from right table (customers). In this join, if there are records in the left table that do not match records in the right table (orders with customers #10 and 11), then MySQL will output records for these orders with NULL values for the customer fields.

mysql8.PNG

Practice: See if you can create joins for the following:

* List the price field with the item_details fields.
* List the price field with the item_details fields, but ordered by order number, not item number.

Another thing we can do with SELECT commands is GROUP BY. We can also use mathematical functions on numeric fields. Try entering the following commands and see if you can figure out what GROUP BY does:


select * from item_details;
select FK_order_no, sum(count) from item_details group by FK_order_no;

We are summing the count fields by order number. So, we can use the GROUP BY clause to group together the values in a given column. We can then use the HAVING clause to exclude groups.

select FK_order_no, sum(count) from item_details group by FK_order_no having sum(count) > 100;

This outputs only the rows where the sum of the counts is greater than 100.
For more on the use of mathematical functions in MySQL, click here. For more on the use of GROUP BY and HAVING, click here.

Practice: Can you list the total amount for each order as well as the sum of the item counts? This requires a join along with GROUP BY.

Here is the solution to the practice exercise. It's important to understand how this works - it illustrates some of MySQL's most powerful features.

mysql9.PNG

Practice: See if you can design a SELECT statement that outputs the three columns above, along with the customer name for each order. Hint: It is possible to have successive joins in a single SELECT statement as in:

select ... from item_details left join items on FK_item_no=item_no left join orders
on FK_order_no=order_no ...

A common need in working with the Ads database system is doing queries on dates. We can add a date field to our orders table to indicate the date the order was placed:

alter table orders add column order_date date;

Then we will populate it like this. Note the date format required for a date type in MySQL

update orders set order_date="2006/11/01";
update orders set order_date="2005/11/01" where order_no < 6;

To do queries, we use comparison operators as in:


select * from orders where order_date < "2006/06/01";

For more information on the SELECT command, check the MySQL Documentation.

Before we leave this section, we should mention that there are other ways of setting up tables. Instead of marking foreign keys with an FK prefix as we did above, it is also common to name the keys the same. For example,

create table customer2 (cust_no int not null auto_increment, name varchar(40),
address varchar(40), phone varchar(12), primary key(cust_no));
create table orders2 (order_no int not null auto_increment, cust_no int not null,
foreign key(cust_no) references customer(cust_no), primary key(order_no)) type=InnoDB;

Then, when you do selects, you need to identify the table name with cust_no, so MySQL knows to which one you are referring.

select * from orders2 inner join customer2 on customer2.cust_no = orders2.cust_no;

Transactions in MySQL

A transaction is a sequence of individual database operations that are grouped together. A transaction will never be complete unless each individual operation within the group is successful. If any operation within the transaction fails, the entire transaction will fail. Transactions are a relatively new addition to MySQL but not to relational database systems in general.

A good example where transactions are useful is in banking. Say you want to transfer $100 between two accounts. In order to deposit money into one account, you must first take money from another account. Without using transactions, you would have to do the following MySQL operations:

1. Check that the balance of the first account is greater than $100.
2. Deduct $100 from the first account.
3. Add $100 to the second account.

If we think of this sequence as a transaction, then if any one operation fails, the whole transaction fails and we rollback, that is, the tables and the data inside them revert to their previous state. If the transaction is successful, we commit the changes to the database. This is much easier than dealing with possible errors between each step. For example, without transactions we need to skip steps #2 and #3 if the balance is less than $100; we need to skip step #3 if for some reason, we were unable to deduct $100 from the first account; and so on. Transactions can simplify the processing.

Transactions have the following four properties, usually referred to by the acronym ACID:

* Atomicity: An atom is meant to be something that cannot be divided. The operations that make up a transaction must either all be carried out, or none at all (as with our banking example).

* Consistency: The database changes state upon a successfully committed transaction.

* Isolation: Data being used for one transaction cannot be used by another transaction until the first transaction is complete. This enables transactions to operate independently of and transparent to each other.

* Durability: This ensures that the result or effect of a committed transaction persists in case of a system failure.

To use transactions in MySQL, you must use a transaction-safe table type. The default MySQL table type, MyISAM, does not support transactions. BerkeleyDB and InnoDB are the transaction-safe table types available in open source MySQL, version 3.23.34 and greater.

Support for InnoDB tables requires a specific compilation parameter when compiling MySQL from source. If your MySQL installation supports InnoDB tables, simply add a "TYPE=InnoDB" definition to the table creation statement. You can check if you have InnoDB support by entering the following command:


show variables like 'have_innodb';

In MySQL, transactions begin with the statement BEGIN WORK or START TRANSACTION and end with either a COMMIT or a ROLLBACK statement. The SQL commands between the beginning and ending statements form the operations of the transaction. Going back to the banking example, the following statements:

update account set balance = balance - 100 where acctnumber = 1;
update account set balance = balance + 100 where acctnumber = 2;

are written as transactions like this:

start transaction;
update account set balance = balance - 100 where acctnumber = 1;
update account set balance = balance + 100 where acctnumber = 2;
commit;

The updates are done in one transaction so that both must complete successfully before the changes are committed to the database. If either update fails, you can issue a rollback statement to undo the changes. For more information on MySQL transactions, refer to the MySQL Documentation.

Finishing Up

When you are ready to leave MySQL, just type "quit". When you come back, just remember that you have to select your database before you can access the tables as in "use plastronics;". You won't need to create the database or the tables again - everything will be there when you return.

One other quick tip: In many MySQL interfaces, you can use the up-arrow key to access a list of commands that you have just entered. This can save a lot of time if you are experimenting with queries.

Additional Examples and Exercises

The following websites have MySQL tutorials, if you need more practice or want to get into more of the details.

MySQL Tutorial

Database Journal MySQL Series

References

MySQL Documentation

Speed Test