Palantir

Work on Hard Problems

Software: A Perpetual Work-in-Progress

Palantir has been building its data integration and analysis platforms (AKA Intelligence Infrastructure) for over six years. Our two products, Palantir Government and Palantir Finance represent a whole new category of software that fundamentally change the way data analysis problems are approached. Our platforms seamlessly integrate disparate sources of data into a single coherent data model. In the end-user realm, our analysis tools sit atop the data platforms, delivering unheard of access, ease-of-use, and security to experts — humans who know a lot about a specific real-world problem but little of data systems.

While it might not be obvious by inspecting any of our shipped versions of our platforms our approach to building products has always been an iterative one. Externally, we always make sure that what gets delivered to users of our software is a complete, usable, and dependable system. Internally, we view the platform as a constantly evolving set of capabilities: each release takes the product the next stable plateau of scale, capability, or performance. From that stable vantage, we learn more about the problems we’re solving and use those lessons to inform the design and constraints of the next iteration of the software.

We’re far from finished with where we’d like to take our platforms and applications – in fact, we think our platforms, being used in production capacities around the world, are in their infancy when compared to what we’ve imagined.

We’re just getting started.

Shepherding Innovation

Our team strikes a careful balance between not re-inventing the wheel and inventing new technology where and when it’s needed. To that end, we’ve been happy to integrate different open source technologies like Lucene, Hadoop, and Cassandra along with industry-standard libraries from Apache and Google to build our software.

When there is no existing, high-quality solution that meets the needs of the platform, we carefully learn about the state of the art from an academic and commercial perspective. Once our team is all on the same page regarding the state-of-the-art, we design, build, and ship new technology designed to meet the unique needs of the users of Palantir’s data platforms.

We first released a production-quality product in 2008. Since then, we have delivered a rapid-succession of features, upgrades, and improvements to both of our platforms.

Customers of Palantir get an ever-evolving (but backwards compatible) set of capabilities as new releases of the platforms are delivered. To get a sense of how a platform evolves, here’s a short list of some of the major features delivered to the Palantir Government platform since its first commercial release (2.0.0 [Hydrogen]) in 2008:

  • Map – The Map application added integrated GIS capabilities to the platform, allowing geospatial analysis, layering, and querying to take place inside the Analyst Workspace, tightly integrated with Palantir’s other capabilities.
  • Nexus Peering – A truly distributed system, Nexus Peering allows a single logical Palantir Government instance be deployed to geographically and net-wise disparate locations. Able to handle unreliable networks, hand-carried offline updates, and employing a practical, low-friction conflict-resolution system, Nexus Peering make information sharing in diffuse organizations a reality.
  • Horizon/Object Explorer – Horizon is a distributed, optimized memory cloud that allows querying across all objects available to a Palantir instance in interactive time. Horizon brings much of the power of Hadoop/MapReduce into an interactive tool. The upshot is that many more, human-directed iterations can occur, leading to nuanced answers to complicated data questions.
  • Palantir Mobile – Going from Hack Week prototype to shipping product in under a year, Palantir Mobile is our Android-platform client, allowing realtime collaboration between personnel in the field and analysts at headquarters.
  • Phoenix – Originally developed by our forward deployed engineering team to meet a specific customer need, Phoenix developed into a product-level offering owned by the product engineering team. Phoenix enables linking a Palantir instance with truly staggering amounts (Peta/Exabyte) of data.

Next Steps