Subscribe to our blog via RSS
Subscribe to our blog via E-Mail

Hard Problems: The Financial Graph

A new vision  

At Addepar we’ve worked with leading financial institutions since the beginning. These early customers were (and continue to be) excellent development partners; like us, they viewed financial technology not for what it is, but what it could be. Together, we re-examined the preconceptions of the investment management world, and invented new ways to represent financial assets and transactions. We want to define a new generation of financial analysis.

The Financial Graph

At the heart of Addepar’s software is the Financial Graph, a new way to represent the financial world. It allows us to perform complex analysis in the simplest, most efficient manner. Storing data in a graph structure—where nodes represent entities and the connections between nodes represent relationships between entities—enables Addepar to quickly determine which pieces of data should be sent through a particular calculation.

So what?

Simply put, the Financial Graph allows you to accurately and efficiently view, report on, and analyze your financial world. “Cash,” for example, is not the same in every account: the liquidity and risk profiles change based on what type of account that cash is in. Cash held in a tax-exempt account is distinct from cash held in a typical brokerage or bank account. Addepar’s graph distinguishes both the aggregated cash position and the positions aggregated by factors like liquidity.

Hard Problems: Transaction Effects

Too many transaction types

The absence of a financial lingua franca forces analysts to spend untold hours compiling and converting disparate data sets in preparation for analysis. In the US alone, there are more than four thousand custodians and trusts, all of them reporting positions in their own way.

A “buy” transaction, for example, might have the transaction code of “BU,” “BUY,” a numeric code, or a code invented by the custodian. Often, custodians use thousands of transaction codes. Alternative assets create further confusion. Funds may report by email, with PDFs, or might still issue paper statements.

Deciphering and standardizing data sets demands a lot of work. Back offices need complex translation and ontology tables to understand their own accounts, and trained analysts capable of more valuable tasks are tied up with what is no more than glorified data entry. 

Simplification is the answer

Addepar treats a transaction’s multiple effects, rather than the transaction alone, as the fundamental unit of financial data. We call these actions Transaction Effects

Transaction Effects accurately represent transactions like those within corporate-action situations or derivative contracts, where any given transaction makes an impact in many different places.

The benefits

Viewing the financial world in terms of all a transaction’s effects allows Addepar to incorporate new and obscure transaction types into its models upon first encounter. Addepar's flexibility saves the user from needing to manually identify, model, and map new transactions. speed and flexibility is key, as typical custodians have hundreds or thousands of unique and proprietary transaction types.

Transaction Effects allow us to maintain a model that accurately represents every transaction, no matter what data provider calls it.

An Atypical Web Vulnerability

A few years ago, Mark Curphey - founder of OWASP - reflected on the state of web security:
"Solving this complex and challenging problem is about having great people, great knowledge, great education, great business process and using great technology.”
Here at Addepar, we believe in a culture where engineers can come together in a collaborative environment to solve some of the most complex and challenging problems facing the financial industry. We work on the bleeding edge of technology and as a result we invariably worry about the security of our platform. No matter how many tests we perform, there is always the worry that there is simply something we didn’t account for. When your environment is constantly undergoing iterative improvements, it is important to recognize that even a few innocently minor issues can cascade into a serious security vulnerability.

In this post, we highlight one such scenario we discovered, while performing a routine internal pentest, where a buffer boundary bug in a third-party framework could have potentially led to serious information leakage. Off the shelf security scanners can only go so far, and as the recent Rails issue have shown, using a popular and commonly used framework does not necessarily protect you.

This is the story of an atypical web vulnerability.

A few months ago, while evaluating different web frameworks, we decided to use LiftWeb for its modularity and built-in security features - Lift is also used by companies such as FourSquare and StackMob. Choosing a solid framework was important to us and Lift's built-in mechanisms against common web application security vulnerabilities - such as CSRF and injection attacks - was an important factor in our decision.

Unfortunately, frameworks are not immune to bugs.

During a routine security audit on one of our rest APIs, we noticed an error message triggered by our web scanner while trying to inject common Cross-Site Scripting (XSS) payloads.

The error message outputted here included the character ‘<’, which suggested that this route might be vulnerable to reflected XSS. We confirmed the finding and the security risk seemed limited due to the target context and the specific Content-Type of the response. It was a simple enough bug but was nevertheless worth reporting and fixing.

We decided to investigate further.

Digging in, we realized that the error message came from Lift’s JsonParser class. Specifically by including an invalid character, such as ‘<’, in the input given to the parser, we were able to trigger an exception whose error message included the content of the input near the invalid character. It is poor practice to blindly expose exceptions across an API boundary but given that we were still actively developing this internal API, we were not surprised to see the error message in the response message.

We were surprised however when, upon further investigation, we realized that simply by increasing the length of our request, we could obtain additional data in the error response . In particular, we were able to retrieve data from other requests.

Looking into the source code, we discovered that the parser used a shared memory buffer. In particular, the input is broken up into chunks that is placed into an in-memory array from which it is parsed. When a parsing error occurs, a 'fail' method (ironic, isn't it?) is called to generate an exception containing characters near the point of failure from the current chunks being parsed.

buf.near is implemented as

The argument to the string constructor is provided in terms of a start and end index, and the math does work out. This looks fairly innocuous, but upon futher examination, it became apparent that the String constructor does not treat the third int argument as a end index but rather as a length. A operation intended to obtain a substring from index 100 to 110 actually obtains the substring from 100 to 210! If the third argument is large enough, it is possible to reach beyond the boundaries of the current chunk being parsed and into the next chunk of data residing in the shared buffer. Since the buffer is shared across requests, the next chunk may or may not belong to the current user session. In effect, this bug enables any authenticated user to retrieve substantial content belonging to other user sessions.

Upon finding this bug, we immediately reported the issue to LiftWeb. In a matter of hours, they fixed the issue on the latest release and within a few days backported the patch to previous versions. Mitre has assigned CVE-2013-3300 for this bug.

Because the boundary violation only occurs when the third argument is sufficiently large, the output of our automated security scanner was not enough to reveal this data leak. It was only through further manual investigation that we discovered the buffer boundary bug and the security impact it had on our API.

Our takeaways from this experience is outlined as follows:

  • Confidence in your code is not enough. Use a solid framework, but make sure it is kept up to date and at least indirectly covered by your test suite. A minor bug in a low level dependency can manifest into a serious security issue.
  • Take care when exposing errors across API boundaries. It might be useful to expose internal errors in a development environment, but this should never be done in a production environment.
  • Always be testing. Manual security testing is important. Automate as much as you can, but do not simply rely on their output. Go deep into every issue. Custom plugins for tools like OWASP ZAP and Burp Suite can make the difference.
  • Check your dependencies for known issues. For Java applications, use OWASP Dependency Check to detect publicly disclosed vulnerabilities in project dependencies. Consider integrating this check into your build tool.
  • Developers and security experts should work together. Identifying the symptoms is not enough. It is much easier to identify root causes, evaluate risks, and determine next steps if there is collaboration.

And yes, we’re hiring!

                                                                                                          Luca Carettoni, InfoSec
                                                                                                          Jerry Zhang, Engineering

Introduction to Hard Problems

Introducing hard problems

Addepar is solving some of the hardest problems in finance. These are problems so embedded in common financial practice that most institutions don’t even recognize them—they were covered up by inefficient, error-prone overstaffing years ago.

We're kicking off a blog series, "Hard Problems," to demonstrate how we're solving some of these issues.

What we do

Addepar forms the cornerstone of  a smart global finance infrastructure. The Addepar platform connects and enhances all your financial data in a single, cloud-based platform. Wealth managers use Addepar to understand, interact with and report on that data for colleagues and clients alike.

We’re unencumbered by antiquated legacy technology. We’ve reimagined financial infrastructure from a clean slate. We’re a company of technologists, in the heart Silicon Valley, who are focused on doing things the right way. Our platform takes advantage of modern technology, leveraging the cloud and the most robust and flexible frameworks to deliver superior aggregation, reporting and analysis software.

Problem: No Common Language

The absence of a financial lingua franca forces analysts to spend untold hours compiling disparate data sets in preparation for analysis. In the US alone, there are more than four thousand custodians, all of them reporting positions in their own way. Understanding the different transaction and how they affect various holdings has traditionally been a manual and error prone process. 

Problem: Error-prone manual input

What’s worse, any data produced by human hands is riddled with missing or even incorrect information. Analysts compiling and manipulating multi-coded data create hard-to-detect errors. Over the course of an investment or a year, these small mistakes can become big problems.

Corrections or edits can take weeks to appear on official bank records. This means that advisors have to settle for bad numbers until reconciliation teams painstakingly fix them. As the system stands now, analysts must—again, manually—double- and triple-check client-facing reports to remove their own errors.

Problem: Unrealized potential

Though management firms spend hundreds of thousands of dollars a year on market research, few automatically incorporate this data into their practice. Typically, common benchmarking data sits unused on the server. If firms do access the data, wealth managers require teams of back-office analysts to show its significance to individual clients.

Problem: False Representation of the Financial World

The current technology fails to offer a clear picture of the financial world. Assets labeled as “cash,” for example, make no indication of various liquidity and risk profiles. Cash held in a tax-exempt account is very different than cash held in a typical brokerage or bank account, and should be presented accordingly.

In addition, traditional technology constrains financial advisors to seeing accounts one at a time, when of course they need a holistic view to do their job well. Clients, too, suffer from this piecemeal approach to wealth management. It’s nearly impossible to deliver a rollup of a multi-asset class portfolio, and even harder to produce comprehensive reports on the performance of a country or sector. 

Lack of flexibility

Wealth advisors often need to customize reports or reclassify data. Something as simple as reclassifying assets, however, requires an IT consultant or hours of manual data entry. Likewise, changing a report template demands weeks of delay and costly customization fees. Advisors find it nearly impossible to add new types of data to the current decade-old software, and analysis tools based on antiquated methodologies bog them down. 

Systems for the lowest common denominator

Finally, yesterday’s technology is designed for the lowest common denominator. In many cases, it’s insufficient or underpowered for sophisticated wealth advisors. Complicated workarounds have to be created for alternative asset classes, complex legal entities and sophisticated tax strategies. In brief, mass-market tools are not built for cutting-edge wealth management.

Addepar: Leadership and Momentum

The global financial crisis in 2008 was crushing on many levels. The series of failures forced us all to reconsider even the most basic assumptions that inform our spending, savings, and investment habits. It pushed market participants of all types to rethink the true risks that underlie assets, and to recalibrate expectations for returns in the “new normal” regime. Each vignette of the crisis tells a different tale, but they remind us that knowledge is power, and ignorance is certainly not bliss.

Joe Lonsdale and Jason Mirra co-founded Addepar in early 2009 with a bold vision: fix finance by building technology that democratizes data to empower investors.

That’s the holy-grail problem. It is intractable without a truly world-class team of developers, data scientists, designers, security gurus, financial analysts, operators and investors who think big, communicate well, and execute consistently. Joe and Jason did a brilliant job attracting like-minded stars, and today Addepar is that team.

Over the past several months, I’ve had the pleasure to meet with dozens of Addepar’s clients and partners. We have an extraordinary constellation of top investors, wealth managers, funds and institutions on Addepar, who have entrusted us to deliver innovative and purpose-built products. The market is clearly embracing our products and vision, and our rapid growth has been fueled in part by the passion we have for solving problems completely with end-users. As we continue liberating more of the world’s financial data, we empower asset owners to do substantially more in far less time with much higher confidence.

We win early and often with clients, and we sustain and fortify our wins with technology. We have exactly the right team to deepen our presence in existing markets, and our unrivaled network is what propels us into adjacent markets on the global stage. The core technology and cloud-based data and analytics platform is what will become the smart infrastructure that enables us all to reimagine global finance.

It’s my great pleasure to welcome Karen White to Addepar’s team, as President and COO. Karen brings 25 years of executive leadership experience, with the common thread of building rock-solid and scalable businesses around next-generation technologies. As a spectacularly well-regarded veteran in Silicon Valley, Karen deeply understands what makes Addepar tick, and she has remarkable insights, impeccable focus, and competitive drive to help build Addepar into the long-lasting institution that it has the potential to become.

Karen and I couldn’t be more thrilled to lock arms with the team that has brought Addepar to the stage we are today, and turn our vision into reality.

Releases at Addepar

Deploying code to your servers can be a huge pain. It usually involves coordinating different moving parts across multiple environments - production, staging, testing, testing-testing, i’m-embarrassed-to-show-anyone-testing, etc. Typically, scripts are written to automate the processes - it is usually some combination of Capistrano, Puppet, and some gnarly shell scripts.

If you’re Twitter, then you have a completely badass deployment system that leverages BitTorrent. But hey, they even called it Murder. As the application evolves, the scripts also need to evolve and as a result they can be fragile. It can be daunting for engineers to get their code up and running on their own testing servers.

We’re not quite at the scale of Twitter and I *imagine* that there might be a few companies that are in the same boat. Etsy has a great post on their deployment tool and process and it was an inspiration for our tool and subsequent blog post.

It is important to have a tool that makes it easy to create and destroy an environment. EC2 makes it easy to add and remove hardware instances, Puppet makes it easy to to create and maintain servers, but there isn’t a clear solution for deploying our application.

We built a tool that makes it easy to create and destroy Addepar’s environment. We set out the following guidelines for our tool

“One Click"
The goal was to make it as seamless as possible and to limit any human involvement with the process of deployment.

Link out deployments with Git
Only committed code can be deployed - it makes it so that we can track anything that is put on our servers.

We only want a few people to have the ability to deploy to production.

We want to be able to track the history of our servers and be able to go back to any arbitrary point in time and see the state of the server - commit hash, the person who deployed, etc.

Since our releases are linked to our source control, we can overlay the git hashes of our releases on our performance metrics. We can automatically track the impact of each of our deployments on CPU performance, memory footprint, etc.

In short, to accomplish our goals we did the following:

1. Build a frontend which lets the user specify the machine and git hash which they want to deploy

2. The frontend makes a request to our Jenkins build server

3. Jenkins runs our Python orchestration scripts, which leverage Fabric

4. Our fabric scripts backup any data, teardown the old environment, and create the new environment.

With our deployment tool, we’ve automated the tedium of deploying simultaneously making it more secure and informative. It has saved us a ton of man hours and has been a huge productivity lever. We’re focusing on the problems that matter.


Addepar emerged from necessity.

The financial crisis of 2008 marked a structural power shift in investment management, not just a cyclical correction. Investors now have the leverage to demand transparency, and expect answers as soon as questions arise. Complete answers, however, remain unavailable from financial giants. Wall Street responded to the 2008 crisis with quick fixes—technological band-aids—rather than building new, more accountable systems. Wealth-management professionals, as a result, rely on incomplete data, in countless different formats, sent from disparate sources.

Financial products and services are the largest market in the world, with over $1 trillion in annual revenue. But, as we’ve seen, the financial infrastructure is broken. It’s about time for finance—the lifeblood of the global economy—to catch up with other sectors and modernize its technology. The new possibilities offered by “big data” will soon replace closed networks, cut out middlemen, and empower investors. Open platforms will give inside deals and fraudulent advisors no place to hide. This is a good thing for honest wealth managers, and a good thing for the world.

Chief Technical Officer Jason Mirra and I founded Addepar in 2009 in response to a broadening chorus of advisors and investors demanding wealth-management software that works. The revolutionary Addepar platform establishes root access to the user’s custodian banks, and assimilates this data along with the figures from all other accounts (including alternative investments) into a single elegant format. With Addepar, wealth-management professionals at any level—from registered investment advisors to advisors at major private banks—have the power to see precisely where they stand, at a glance, at any given moment.

We developed Addepar in close partnership with some of the world’s most successful investors, and recruited top Silicon Valley software engineers to tackle the challenges they defined. Addepar’s visionary data management removes the need for manual retrieval, manual entry, and multi-format PDFs. It saves time, prevents costly mistakes, and motivates better decision making.

Best of all, Addepar has the power to evolve. Addepar’s open architecture allows our engineers, as well as the users, to customize workflows and automate new tasks as they materialize. The platform has the flexibility to cope with the ever-changing landscape of financial data, while remaining intuitive to users across the spectrum of wealth management.

Addepar has grown into a company with over 80 full-time employees and offices in Silicon Valley and New York City, with more on the way. Our client base includes family offices, multi-family offices, RIAs, private banks, and fund of funds. We aim to make Addepar the standard software for every level of wealth management. Our clients love us.

Today, we continue to attract the best and the brightest to help us transform global finance. Our mission is just beginning.

Respectfully yours,

Joe Lonsdale
Co-founder and Executive Chairman