2010: SCHISM: a Workload-driven Approach to Database Replication and Partitionoing

2010: “Schism: a Workload-Driven Approach to Database Replication and Partitioning”, Carlo Curino, Yang Zhang, Evan Jones, Sam Madden, accepted for publication to Proceedings of Very Large Data Base (VLDB)

ABSTRACT:

We present Schism, a novel workload-aware approach for database partitioning and replication designed to improve scalability of shared-nothing distributed databases.  Because distributed transactions are expensive in  OLTP settings (a fact we demonstrate through a series of experiments), our partitioner attempts to minimize the  number of distributed transactions, while producing balanced  partitions. Schism consists of two phases: i) a  workload-driven, graph-based replication/partitioning phase and ii)  an explanation and validation phase. The first phase creates a  graph with a node per tuple (or group of tuples) and edges between  nodes accessed by the same transaction, and then uses a graph  partitioner to split the graph into k balanced partitions that  minimize the number of cross-partition transactions. The second  phase exploits machine learning techniques to find a predicate-based  explanation of the partitioning strategy (i.e., a set of range  predicates that represent the same replication/partitioning scheme  produced by the partitioner).

The strengths of Schism are: i) independence from the schema  layout, ii) effectiveness on n-to-n relations, typical in social  network databases, iii) a unified and fine-grained approach to  replication and partitioning. We implemented and tested a prototype  of Schism on a wide spectrum of test cases, ranging from classical  OLTP workloads (e.g., TPC-C and TPC-E), to more complex scenarios  derived from social network websites (e.g., Epinions.com), whose  schema contains multiple n-to-n relationships, which are known to be  hard to partition. Schism consistently outperforms simple  partitioning schemes, and in some cases proves superior to the best known manual partitioning, reducing the cost of distributed transactions up to 30%.

 

Contact me to receive a copy of the paper.

 

 

Facebook pissed off people… and the open alternatives start to appear

As many have already noticed Facebook hit the mark of too-big.. that makes people go from “that the cool new hype” to “i don’t trust this big brother huge corporation”… if you add the horrible privacy policy and the “i own you” attitude of Facebook… well no surprise new alternatives appear to the horizon… I think an interesting idea is the one of Diaspora: http://www.joindiaspora.com/project.html

They are just at the beginning but the idea of having more control on your own data… well it’s appealing… best of luck to the guys at Diaspora….

 

How to check the consumption state of a manual-gear clutch.

When buying a used car you should check the clutch is ok… here is a mechanics-trick on how to do it.

 

to check the state of a clutch you can:

  1. pull-up the hand break completely
  2. put the gear-lever to a high-gear (e.g., 4)
  3. bring the engine high-up (as if you want to burn a light)
  4. release gently the clutch

Check the following:

  1. A good clutch will not slide and the result is that the car engine will shut-off (since the hand break is holding the car still, and the high gear is disadvantageous for the engine).
  2. A bad clutch will start sliding, the engine do not stop, the car doesn’t move, and you soon will smell dead-clutch.

In either case interrupt the experiment right-away (to avoid to further screw the poor clutch).

 

Enjoy your new car

 

How to check the consumption state of a manual-gear clutch.

When buying a used car you should check the clutch is ok… here is a mechanics-trick on how to do it.

 

to check the state of a clutch you can:

  1. pull-up the hand break completely
  2. put the gear-lever to a high-gear (e.g., 4)
  3. bring the engine high-up (as if you want to burn a light)
  4. release gently the clutch

Check the following:

  1. A good clutch will not slide and the result is that the car engine will shut-off (since the hand break is holding the car still, and the high gear is disadvantageous for the engine).
  2. A bad clutch will start sliding, the engine do not stop, the car doesn’t move, and you soon will smell dead-clutch.

In either case interrupt the experiment right-away (to avoid to further screw the poor clutch).

 

Enjoy your new car

 

2010: Scalable Architecture and Query Optimization for Transaction-time DBs with Evolving Schemas

Title: Scalable Architecture and Query Optimization for Transaction-time DBs with Evolving Schemas

Venue: SIGMOD 2010

Abstract:

The problem of archiving and querying the history of a database is made more complex by the fact that, along with the database content, the database schema also evolves with time. Indeed, archival quality can only be guaranteed by storing past database contents using the schema versions under which they were originally created. This causes major usability and scalability problems in preservation,retrieval and querying of databases with intense evolution histories, i.e., hundreds of schema versions.This scenarios are common in web information systems and scientific databases that frequently accumulate that many versions in just few years.

Our system, Archival Information Management System (AIMS), solves this usability issue by letting users write queries against a chosen schema version and then performing for the users the rewriting and execution of queries on all the appropriate schema versions. AIMS achieves scalability by using (i) and advanced storage strategy based on relational technology and attribute-level-timestamping of the history of the database content, (ii) suitable temporal indexing and clustering techniques, and (iii) novel temporal query optimizations. In particular, with AIMS we introduce a novel technique called CoalNesT that achieves unprecedented performance when temporal coalescing tuples fragmented by schema changes.Extensive experiments show that the performance and scalability thus achieved greatly exceeds those obtained by previous approaches. The AIMS technology is easily deployed by plugging into existing DBMS replication technologies, leading to very low overhead;moreover, by decoupling logical and physical layer provides multiple query interfaces, from the basic archive&query features considered in the upcoming SQL standards, to the much richer XML XQuery capabilities proposed by temporal database researchers.

 

More information available at Panta Rhei: Schema Evolution and Temporal Database Tools

 

Database in the Cloud

The advent of cloud computing and hosted software as a service is creating a novel market for data management. Cloud-based DB services are starting to appear, and have the potential to attract customers from very diverse sectors of the market, from small businesses aiming at reducing the total cost of ownership (perfectly suited for multi-tenancy solutions), to very large enterprises (seeking high-profile solutions spanning on potentially thousands of machines and capable of absorbing unexpected burst of traffic). At the same time, the database community is producing new architectures and systems tackling in novel ways various classes of typical DB workloads. These novel, dedicated approaches often outshine the traditional DBMS trying to provide an unreachable “one-size-fit-all” dream.  Such increase in the number of available data management products is exacerbating the problem of selecting, deploying and tuning in-house solutions for data management.  

In this context we are building a cloud-based database architecture, enabling the implementation of DB-as-a-service.  We envision a database provider that (i) provides the illusion of infinite resources, by continuously meeting user expectations under evolving workloads, and (ii) minimizes the operational costs associated to each user by amortizing administrative costs across users and developing software techniques to automate the management of many databases running inside of a single data center.  Thus, the traditional provisioning problem (what resources to allocate for a single database) becomes an optimization issue, where a large user base, multiple DBMS engines, and a very large data center provide an unprecedented opportunity to exploit economy of scale, smart load balancing, higher power efficiency, and principled overselling.

The “relational cloud” infrastructure has several strong points for database users:  (i) predictable and lower costs, proportional to the quality of service and actual workloads, (ii) reduced technical complexity, thanks to a unified access interface and the delegation of DB tuning and administration, and (iii) elasticity and scalability, providing the perception of virtually infinite resources ready at hand. In order to achieve this, we are working on harnessing many recent technological advances in data management by efficiently exploiting multiple DBMS engines (targeting different types of workloads) in a self-balancing solution that optimizes the assignment of resources of very large data centers to serve potentially thousands of users with very diverse needs.  Among the critical research challenges to be faced we have identified: (i) automatic database partitioning, (ii) multi-db, multi-engine workload balancing, (iii) scalable multi-tenancy strategies, (iv) high profile distributed transaction support, (v) automated replication and backup.   These and many others are the key ingredients of the RelationalCloud (http://relationalcloud.com) prototype currently under development.

XEN on OpenSolaris step-by-step

This article present is a webdiary of the installation of a XEN-based virtualized server. The Dom0 of choice is OpenSolaris, since it provides the nifty features of ZFS.

We are buying a fairly powerful machine, a:

 – 2xQuadcore XEON 2.2.6 Ghz 8MB cache

 – 24 GB RAM

 – 6 x 1TB Seagate 7200 RPM hot-swap SATA disks  (32MB cache)

This will be soon installed with OpenSolaris and we will try to virtualize with XEN, to host DomU with ubuntu (old and new), and maybe a Windows VM. 

 

 

2009 SIGMOD 09: “PRIMA: Querying Historical Data with Evolving Schemas”

 PRIMA: Archiving and Querying Historical Data with Evolving SchemasHyun J. Moon, Carlo A. Curino,  MyungWon Ham, Carlo Zaniolo, accepted as demo paper at International Conference on Management of Data ’09 (SIGMOD)

ABSTRACT:

Schema evolution poses serious challenges in historical data management. Traditionally the archival data has been (i) either migrated under the current schema version, to ease querying, but compromising archival quality, or (ii) maintained under the original schema version in which they firstly appeared, leading to a perfect archival quality, but to a taxing query interface.
The PRIMA system, we present, achieves the best of both worlds, by archiving data under the original schema version, while automatically adapting the user temporal queries to the appropriate schema versions. The user is entitled to query the archive under a schema version of choice, letting the system to rewrite the queries to the potentially many involved schema versions. Moreover, the system offers automatic documentation of the schema history, and allows to pose temporal queries over the metadata history itself.
The proposed demonstration, highlights the system features exploiting both a synthetic-educational running example and the real-life evolution histories (schemas and data).
The selected real-life systems include, but are not limited to, the popular genomic database Ensembl and of Wikipedia, with their hundreds of schema versions.
The demonstration offers a thorough walk through the system features and an hands-on system testing phase, in which the audience is invited to interact directly with the advanced query interface of PRIMA. The conference participants will freely pose complex temporal queries over transaction-time databases subject to schema evolution, observing PRIMA rewriting and query execution capabilities.