When we first started developing our data fusion platforms, we wanted to create a way to integrate data for secure, collaborative analysis at scale. We knew that achieving this outcome would not be a matter of simply making more data available to more analysts. We wanted to give our users a way to learn from and build on not just the raw data in their enterprise, but the insights of their colleagues, too. The right technology had to integrate both data and analysis.
As we grew and deployed our technology to more (and more diverse) users, we saw first-hand how collaborative data analysis can save time, save money, and even save lives. We also saw a lot of unrealized potential. If the advantages were so apparent, we wondered, why didn’t we observe more collaborative data analysis between organizations where policy allowed it—allied nations, neighboring public institutions, and partner commercial enterprises?
In turns out that integrating data for secure, collaborative analysis isn’t as easy as throwing everything onto a share drive. Many of our customers wanted to collaborate more, but were prevented from doing so by several hard, multi-faceted technical problems:
Nexus Peering is Palantir’s solution to these problems. Nexus Peering enables information-sharing at the institutional level, allowing teams, agencies, and governments to exchange data and analysis in almost any direction or environment while maintaining consistency, integrity, and security.
So how does Nexus Peering work? A comprehensive explanation would require more than a simple blog post. But for introductory purposes, you can think of each installation or instance of our Palantir Gotham data fusion platform as maintaining its own “nexus” of data. Nexuses can incorporate changes made by users at different Palantir Gotham instances through acts of synchronization, or “peering.” The main goal of Nexus Peering is to ensure that data is always in a consistent state across instances. To this end, Nexus Peering must capture, circulate, and merge changes to shared data while recognizing and resolving data conflicts.
Each nexus maintains a record of each change to every piece of its data in a manner similar to how revision control software such as Git or Mercurial tracks changes to codebases during software development. Upon log-in, each user “checks out” a copy of the consensus view of the data (known as the Base Realm) and works with it on a private branch. When a user wishes to share the results of his or her analysis with the rest of the team, he or she publishes the changes back to the Base Realm.
Nexus Peering keeps track of these changes across nexuses, even in cases where the peered instances are not in constant communication. In order to merge changes that were made to the data concurrently by users at different nexuses, Nexus Peering uses a technique called ‘version vectors’ to keep track of and automatically apply the changes in the proper order when peering. In cases when concurrent changes conflict, users are alerted and forced to determine for themselves which version of the data they want to work with.
That’s a brief overview of how Nexus Peering works. For an in-depth explanation, check out the presentation from GovCon 6 below.
Late in 2012, we upgraded the instances on our largest Nexus Peering mesh network (spanning 40 locations and four continents) to Palantir Gotham 3.8, which includes a host of new Nexus Peering features. Here are some highlights:
Today, Nexus Peering works across data models, security models, time zones, and borders. It does not require all data to be shared and it does not require an active connection between the two nexuses being peered. Nexus Peering captures and circulates changes between nexuses, while recognizing and resolving conflicting changes, all without destroying or duplicating any data. With Nexus Peering, institutions can create a unified, shared, enterprise-wide information picture that is updated at whatever frequency is possible. Sometimes that’s as real-time as a high-speed network will allow, sometimes it’s as fast as the physical drive containing the latest updates can be hand-delivered.
This is what a complex Nexus Peering network might look like. Each box represents a Palantir Gotham instance, or “nexus.” (Place names chosen at random from a Diplomacy board.)
Based on the technical improvements introduced in the latest version, we envision even more compelling use cases for Nexus Peering. Cooperation between different organizations, including international partners, is now significantly easier, enabling unprecedented secure collaboration on common threats such as cyber attacks. The same security frameworks that allow governments to share data can now also allow secure sharing between the private sector and government, while protecting individual privacy and organizational sensitivities. New use cases could include banks sharing cyber threat data with law enforcement, pharmaceutical companies sharing product safety data with regulatory agencies, peering between agencies involved in disease control, and peering between our philanthropic partners. We’re looking forward to seeing what the future of Nexus Peering holds.