I want to show that the notion of scalability is every bit as valid when applied to programming languages as it is when applied to programs or algorithms. I'll also discuss several well-known and not so well-known programming languages from this perspective and give some concrete recommendations, as well as discuss some of the social factors which hinder progress in this field.
Red Hat on Wednesday announced a significant departure from its current business plan, saying its flagship Linux product will be available on Amazon.com's Elastic Computing Cloud online service.
Google Maps, Yahoo! Mail, Facebook, MySpace, YouTube, and Amazon are examples of Web sites built to scale. They access petabytes of data sending terabits per second to millions of users worldwide. The magnitude is awe-inspiring. Users view these large-scale Web sites from a narrower perspective. The typical user has megabytes of data that are downloaded at a few hundred kilobits per second. Users are not so interested in the massive number of requests per second being served; they care more about their individual requests. As they use these Web applications, they inevitably ask the same question: "Why is this site so slow?" The answer hinges on where development teams focus their performance improvements. Performance for the sake of scalability is rightly focused on the back end. Database tuning, replicating architectures, customized data caching, and so on allow Web servers to handle a greater number of requests.
James Hamilton has published a thorough summary of Facebook's Cassandra, another scalable key-value store for your perusal. It's open source and is described as a "BigTable data model running on a Dynamo-like infrastructure." Cassandra is used in Facebook as an email search system containing 25TB and over 100m mailboxes. # Google Code for Cassandra - A Structured Storage System on a P2P Network # SIGMOD 2008 Presentation. # Video Presentation at Facebook # Facebook Engineering Blog for Cassandra # Anti-RDBMS: A list of distributed key-value stores # Facebook Cassandra Architecture and Design by James Hamilton
February 17th, 2008 A 'problem' with the Mongrel/Rails platform A request comes in to the web server, and if it's dynamic, falls down to a waiting mongrel process. Mongrel calls Rails which wraps a big lock around most of the request/response cycle so the mongrel thats serving the request has to wait for a response from Rails to unlock and start serving other requests. This is the Blocked Thread stability anti-pattern that Michael Nygard talks about in Release It!. The problem is that Nginx or Apache doesn't know that a mongrel is blocked and keeps blindly sending requests. Thats means that even a single blocked mongrel will result in some slow responses for requests that come in to that same mongrel. Cut the Request Fat And one way to do that is to use the "Skinny Request, Fat Backend" principle. What that means in practical terms is pretty simple: Do as little as possible inside of the Request/Response cycle.
I did some research recently on memcached and how it compares to ehcache. The following graph shows the time taken for 10,000 puts, gets and removes, for 10,000 cache items. It uses the latest released versions of memcached and ehcache. In memcached's cas
A collection of Concurrent and Highly Scalable Utilities. These are intended as direct replacements for the java.util.* or java.util.concurrent.* collections but with better performance when many CPUs are using the collection concurrently.
There's a lot of buzz around the Ruby on Rails framework at the moment. Something that doesn't get a lot of attention is it's scalability solution: FastCGI. Partly because it doesn't seem very exciting on first glance and partly because it's so fundamenta
Red Hat Cluster Suite is a collection of technologies working together to provide data integrity and the ability to maintain application availability in the event of a failure. Administrators can deploy enterprise cluster solutions using a combination of hardware redundancy along with the failover and load-balancing technologies in Red Hat Cluster Suite.
create account or login Content Home Download License Documentation Whitepapers Test results FAQ Community Source code Join the mailing list Mailing list archives Bug tracker Guidelines Contribute FastMQ Partners Services Contact Tags amqp broker brokerless c community design distributed exchange gpl java kernel latency license market matching methodology optimisation perf proximity python queue speed standards testing throughput trading tutorial Copyright (c) 2007-2008 FastMQ Inc. Licensed under the Creative Commons Attribution-Share Alike 3.0 License. Fastest. Messaging. Ever. What's messaging? Getting two apps to talk is always a challenge. They have to agree on what the data looks like. And what it means. They have to agree on how the data is organised into a message. XML, binary, or something else? They have to agree how to speak to each other. They have to agree on security. They have to agree how to connect, across what networks. They need to queue messages that can't be
mainly marketing articlle by cofiiunder What if you didn't have to do any of this funny business to get scalability and reliability? What if the JVM had access to a service that you could plug into to make its heap durable, arbitrarily large, and shared with every other JVM in your application tier? Enter Terracotta, network-attached, durable virtual heap for the JVM. In the spirit of full-disclosure, I'm a co-founder of Terracotta and work there as a software developer. Terracotta is an infrastructure service that is deployed as a stand-alone server plus a library that plugs into your existing JVMs and transparently clusters your JVM's heap. Terracotta makes some of your JVM heap shared via a network connection to the Terracotta server so that a bunch of JVMs can all access the shared heap as if it were local heap. You can think of it like a network-attached filesystem, but for your object data; see Figure 1.
Violin Memory has launched the industry's first terabyte-scale Memory Appliance, the Violin 1010. Memory Appliances are flexible and scalable memory systems that can be used as either Extended Memory or High Performance Storage.
In early 2006, Marshall University laid out a plan to migrate HOBBIT (Figure 1), an HP OpenVMS cluster handling university-wide e-mail services. Plagued with increasing spam attacks, this cluster experienced severe performance degradation. Although our employee e-mail store was moved to Microsoft Exchange in recent years, e-mail routing, mailing list and student e-mail store (including IMAP and POP3 services) were still served by OpenVMS with about 30,000 active users. HOBBIT's e-mail software, PMDF, provided a rather limited feature set while charging a high licensing fee. A major bottleneck was discovered on its external disk storage system: the dated storage technology resulted in a limited disk I/O throughput (40MB/second at maximal) in an e-mail system doing intensive I/O operations.
Google Tech Talks June 23, 2007 ABSTRACT This talk will discuss some of the scalability challenges that have arisen during YouTube's short but extraordinary history. YouTube has grown incredibly rapidly despite having had only a handful of people responsible for scaling the site. Topics of discussion will include hardware scalability, software scalability, and database scalability. Speaker: Cuong Do Cuong is currently an engineering manager at YouTube/Google. He was part of the engineering team that scaled the YouTube software and hardware infrastructure from its infancy to its current scale. Prior to YouTube/Google, he held various software development and management positions at PayPal and Inktomi.
Facebook revealed some big, big stats on big data to a few reporters at its HQ today, including that its system processes 2.5 billion pieces of content and 500+ terabytes of data each day. It’s pulling in 2.7 billion Like actions and 300 million photos per day, and it scans roughly 105 terabytes of data each half hour. Plus it gave the first details on its new “Project Prism”.
Note that I made a grave error of thinking before writing this article: I forgot the copy-on-write page sharing of modern Unices. I added two paragraphs to this article that should clarify the point. Thanks for your comment, Alex.
Jazzmaster is an application framework developed by Eiichiro Uchiumi (Eiichiro.org) and can be performed on JDK 5 or later Java platform. Using Jazzmaster framework, you can construct the application from the small one which is processed on a command line to the large one like enterprise application on the paradigm of SOA (Service-Oriented Architecture). The application which is built by composing the immediately-modularized services on the Jazzmaster framework certainly improves the flexibility of changeability and the development agility, according to becoming if the scale grows.
Jazzmaster is composed of the following modules:
* Core
* AspectJ integration
High-performance Linux clustering, Part 1: Clustering fundamentals Introducing the basic concepts of High Performance Computing with Linux cluster technology
H. Wang, K. Zhang, Q. Liu, D. Tran, und Y. Yu. Proceedings of the 5th European Semantic Web Conference, Berlin, Heidelberg, Springer Verlag, (Juni 2008)
D. Abadi, A. Marcus, S. Madden, und K. Hollenbach. VLDB '07: Proceedings of the 33rd international conference on Very large data bases, Seite 411--422. VLDB Endowment, (2007)