Talent Migration and The Open Source Revolution

While headlines are dominated by talk of immigration and border walls, a different kind of migration among the tech community is creating a revolution in the world of open source. Where innovation was once closely guarded and major advances in data aggregation analysis was led by only a few large companies, we are now finding that how we leverage online data is fundamentally changing thanks to increasingly open collaboration and movement between people and networks. Few areas have seen more of a step-change evolution thanks to open source than social graphs, where the development and open source cultivation of Giraph has opened up groundbreaking opportunities for companies of all sizes.

Not long ago, the very best ideas were patented and protected at all costs. Closed doors and high secrecy ensured that a small cadre of larger companies could hoard their very best ideas and eliminate any real or perceived threat from competitors. In today’s increasingly networked world, patents are being supplanted by open source solutions, where communities can iterate collectively to improve experiences, rather than rely on the few hands of developers behind closed doors.

The reality of open source is that it outlives any single company, and at the core of this open source revolution is the migration of talented people. Whereas Google created some of the world’s best data infrastructure only a few years ago, they shared few if any details with the rest of the tech community. However, as talented employees moved on to other companies like Facebook, they brought their knowledge with them, creating analogous – but incrementally improved – platforms, then open sourcing their efforts. Beyond Facebook, with the emergence of AirBnB, Yelp, Uber, and many others, the migration of talent has inevitably also led to the migration (and evolution) of ideas. And so, whether Google liked it or not, open source became the norm, strengthening companies and consumers alike.

There are few better examples of open source benefitting all stakeholders than the creation and evolution of Giraph. For companies once relying on Hadoop to run expensive computations on their networks, Giraph represents a far more specialized, nimble, and expedient alternative. If Hadoop is a hammer, Giraph is a specialized #2 Phillips head screwdriver. The efficiencies of Giraph have enabled folks to run algorithms on networks with trillions of data points in minutes rather than weeks.

In the case of Giraph, the benefits of open source are obvious. Today, at the heart of every major tech company’s inherent value is its own core network of data and the interplay between each individual point of data. If you rip out Facebook’s social graph, the platform becomes essentially useless. The same goes for Yelp’s network of restaurants, Uber’s network of drivers, and any other online service that aggregates complex data sets. So the creation and continuous improvement of such a platform will enable these companies to target customers in increasingly effective, sophisticated, and meaningful ways.

Perhaps most exciting about the evolution of social graphing through Giraph is that it demonstrates that, despite these massive advances, we are still in the infant stages of open source. Consider that only a few years ago, Google was holding onto game-changing innovations that are now broadly shared and improved upon throughout the tech community. As the migration of talent continues to spread, so too will the value, performance, and sophistication of these platforms, empowering companies and consumers well beyond platforms like Giraph, in ways that we can only now imagine.

Nitay Joffe

Nitay Joffe

Nitay Joffe is founder and CTO at ActionIQ. He started ActionIQ to explore his passion for innovations in databases, distributed systems, and big data. He was an instrumental engineer in Facebook’s data infrastructure initiatives, and a core contributor to open source projects HBase and Giraph.
Nitay Joffe
Return to Blog

Are you ready to unlock your customer data?