Making web sites is not programming!

I know a lot of people who claim they can make web sites. Not all software is good software. Learn your basics!

ENGINEERING = SIMPLE SOLUTIONS WITH MINIMAL COMPLEXITY

A substantial number of software engineering problems posed in the past 20 years have been optimally solved using the simplest solutions (with minimal complexity). De-allocation of unused pointers in code, for instance, is one of the main issues in code optimization. This procedure has provoked many newer generation programming languages in their attempt to automatize it. All these issues were (and still are) very important factors to consider in most types of software application development. However, there is one branch of software engineering that compromises the principle of simple solutions.

ENTER WEB DEVELOPMENT!

Together with mobile development, web application development is one of the newer branches of informatics. But web application development has gone a long way since those static Internet HTML pages, which used to be the epitomy of a web site. Today, most web applications run faster, more reliably and are more practical to use than their desktop counterparts. Furthermore, a web site (not to be confused with a web application) is advertised as being one of the easiest software to build, thanks to all the fancy new automation out there on the web, which can assist literally anyone, in starting a new Internet presence. This is precisely where all the problems begin!

AUTOMATION IS NOT ALWAYS A BETTER SOLUTION

Namely, anybody with the ability to type www.godaddy.com on their browser for example, is a potential web developer (by using their built in automatic script and page generators). The result of this is a sea of boring, monotonic badly engineered templated web sites popping up on the web.

An even bigger problem is when these sites start offering services that require the user to enter credit card details. These abominations of software coding are deemed development detrimental to the reputation of software engineering in general. First, because of the quality of the “build”, secondly because of the authors (usually without any formal knowledge in it, or software engineering) bragging about how easy it is to create a web application. Hence, anyone can make web sites (sometimes erroneously used interchangeably with web application) !?

CODING IS A PRECISE SCIENCE

You want to write a web application or put up a web site? Cool. Ask a developer to help you write the code and a designer to make it appealing to look at. Or here’s another novel idea: don’t! Nobody needs to see more of whatever temporal hogwash you think qualifies as being a subject of artistic expression. For the love of humanity, it is a precise science, which requires meticulous planning and, in most cases, a team!

HOW TO CREATE A WEBSITE OR WEB APPLICATION

Now for a proposed solution. It is my belief that web applications lack a precise formal definition of three things required to create a website or web application (even the simplest one):

• Planning and design
• Coding and
• Content

Planning and design is the responsibility of the designer. Namely, the person with a keen eye for geometry and color, who should know beforehand what the basic topic and target audience of the website is. The graphic designer comes up with the look and feel of the product and will essentially use tools like Gimp or Photoshop for the graphics, as well as HTML and CSS for the structure, element positioning and cross browser/device compatibility.

Coding falls under the responsibility of the developer. This includes anything where a variable must be declared for instance, backend server side coding, JavaScript, database interoperability and even CSS.

Finally content. That’s all you, the guy or gal wanting to have a web site on the Internet! If you happened to be the same person as the coder or the designer then yes, you can merge into one of the previously mentioned category of responsibilities. Indeed, since you are the person with the idea, you do by all means have a say in how every part of the site looks or feels like, but ultimately when in doubt ask for a new design or a new implementation. If at first, second, or third, this does not work… change your designer or developer!

In any case, just acknowledging distinction above will help in multiple spheres. Firstly, the final product will actually be eye-worthy and internet-worthy (at least in terms on design and implementation). Secondly, web sites and applications like these are much easier to maintain in the long run and can easily be readjusted to comply with new standards in software engineering as needed. Finally, it will bring web application development back to where it belongs in terms of reputation and credibility: web development is engineering, to be more precise software engineering.

Relational or non relational databases? That is the question.

We’ve all been there. Every system designer comes across one question…and every database (DB) admin is faced with consequences from bad decisions. Namely, how do we decide on using databases, which types should we go with, and how can we make sure we make the best of what we have by optimization? Here, we’ll review and answer these questions, and invite your feedback at the end. Let us know what works FOR YOU in terms of database programming and share your tips or experience in choosing DB structures in the comments section.

WHAT IS A DATABASE?

Put in simple terms, databases are nothing more than data containers. They organize data in datasets of rows and columns. They also give the user the ability to search for particular data based on different properties of data. Databases come in two major denominations: relational and non-relational. Going into definition details for each of these is way beyond the scope of this blog post, so we will only mention the primary differences.

1. Relational databases. A relational database manipulates only tables and the result of all operations are also tables. The tables are sets, which are themselves sets of rows and columns. You can view the database itself as a set of tables.

2. Non-relational databases. The non-relational flavor of databases, on the other hand, are based on a “navigational” model: a hierarchy, a linked list, a B-Tree, etc. It’s common to refer to these as ISAM (Indexed Sequential Access Method) Databases.

YOU NEED A DATABASE. SO, WHAT NEXT?

So, having concluded a definite need for a database how do we chose the best one for our applications? Well, first of all it mostly depends on the nature of the information you will store, as well as how it will be accessed.

Store inventories work great as relational databases. For example, managing a store inventory list would be a perfect (and a very simple) example of when to use a relational model DB. You would have a table of items offered by the store, a second table of stores available in the area and a final table which would keep info on the (in-store) quantity of items in each store. Essentially, this is a normalized database design, meaning that each piece of unique data is stored only once and then referenced as needed via an index. The important thing to keep in mind here (in this type of setup) is that the data is limited to how much it can grow, meaning that there is a limited number of stores as well as a limited number of items these stores can offer their customers. Hence, the choice of a relational database is quite a sound choice.

Websites do better with non-relational databases. Now when it comes to the internet, the web introduces a new scale for applications in terms of:

• Concurrent users (millions of reqs/second)
• Data (petabytes generated daily ( think google, facebook, etc.)
• Processing (all this data needs processing)
• Exponential growth (surging unpredictable demands)

Namely, web sites with very large traffic have no way to deal with these issues using existing relational database systems, even using those high end query crunchers like Oracle, MS SQL, Sybase, etc. The reason for this is simple. Related data is distributed amongst different tables which need to be joined in order to fetch that data. These joins perform more slowly the larger the datasets grow. Furthermore, distributing this data to multiple servers makes the joining of related data even slower to fetch. Sure, you can throw all your data to a single table and repeat if need be, but that kinda defeats the point of a relational database system. Popular web sites that have faced these sorts of issues include Google, Yahoo, Facebook, Twitter, Amazon, etc. Basically, web sites dealing with high traffic, massive data, large user base and user generated content.

Compared to other large scale systems such as telco applications, the above mentioned are FREE apps and can therefore compromise on data integrity and consistency. To be blunt, they wont be sued if someone hasn’t received:

his friends status update on time (FB, Twitter) or
the desired result of a search (google)
The solution for handling this type of web scaling lies in non relational database systems. Although there are a number of systems available freely for download (Redis, CouchDB, MongoDB) the top tier companies like Google, Facebook and Amazon went ahead and built their own custom systems (optimized for their own services) from scratch namely, BigTable, Cassandra and Dynamo respectively. The benefits of these systems include the following:

• They are massively scalable
• Highly available, decentralized and don’t have a single point of failure
• They employ transparent sharding
• Parallel processing
• Built in mechanisms for automatic conflict resolution

However, these systems do come with a price, which was proven by scientists at MIT in 2002. To wit, non relational systems must compromise on at least one of the following optimizations: Consistency, Availability and/or Partition tolerance. Under no circumstances can non-relational databases be optimized for all three of the above benefits.

On one of my next "programming tips of the week", I will go deeper into real world examples of specific optimization methods given various types of problems requireg databases are their foundations.