A few years ago, Mark Curphey - founder of OWASP - reflected on the state of web security:
"Solving this complex and challenging problem is about having great people, great knowledge, great education, great business process and using great technology.”Here at Addepar, we believe in a culture where engineers can come together in a collaborative environment to solve some of the most complex and challenging problems facing the financial industry. We work on the bleeding edge of technology and as a result we invariably worry about the security of our platform. No matter how many tests we perform, there is always the worry that there is simply something we didn’t account for. When your environment is constantly undergoing iterative improvements, it is important to recognize that even a few innocently minor issues can cascade into a serious security vulnerability.
In this post, we highlight one such scenario we discovered, while performing a routine internal pentest, where a buffer boundary bug in a third-party framework could have potentially led to serious information leakage. Off the shelf security scanners can only go so far, and as the recent Rails issue have shown, using a popular and commonly used framework does not necessarily protect you.
This is the story of an atypical web vulnerability.
A few months ago, while evaluating different web frameworks, we decided to use LiftWeb for its modularity and built-in security features - Lift is also used by companies such as FourSquare and StackMob. Choosing a solid framework was important to us and Lift's built-in mechanisms against common web application security vulnerabilities - such as CSRF and injection attacks - was an important factor in our decision.
Unfortunately, frameworks are not immune to bugs.
During a routine security audit on one of our rest APIs, we noticed an error message triggered by our web scanner while trying to inject common Cross-Site Scripting (XSS) payloads.
The error message outputted here included the character ‘<’, which suggested that this route might be vulnerable to reflected XSS. We confirmed the finding and the security risk seemed limited due to the target context and the specific Content-Type of the response. It was a simple enough bug but was nevertheless worth reporting and fixing.
We decided to investigate further.
Digging in, we realized that the error message came from Lift’s JsonParser class. Specifically by including an invalid character, such as ‘<’, in the input given to the parser, we were able to trigger an exception whose error message included the content of the input near the invalid character. It is poor practice to blindly expose exceptions across an API boundary but given that we were still actively developing this internal API, we were not surprised to see the error message in the response message.
We were surprised however when, upon further investigation, we realized that simply by increasing the length of our request, we could obtain additional data in the error response . In particular, we were able to retrieve data from other requests.
Looking into the source code, we discovered that the parser used a shared memory buffer. In particular, the input is broken up into chunks that is placed into an in-memory array from which it is parsed. When a parsing error occurs, a 'fail' method (ironic, isn't it?) is called to generate an exception containing characters near the point of failure from the current chunks being parsed.
buf.near is implemented as
The argument to the string constructor is provided in terms of a start and end index, and the math does work out. This looks fairly innocuous, but upon futher examination, it became apparent that the String constructor does not treat the third int argument as a end index but rather as a length. A operation intended to obtain a substring from index 100 to 110 actually obtains the substring from 100 to 210! If the third argument is large enough, it is possible to reach beyond the boundaries of the current chunk being parsed and into the next chunk of data residing in the shared buffer. Since the buffer is shared across requests, the next chunk may or may not belong to the current user session. In effect, this bug enables any authenticated user to retrieve substantial content belonging to other user sessions.
Upon finding this bug, we immediately reported the issue to LiftWeb. In a matter of hours, they fixed the issue on the latest release and within a few days backported the patch to previous versions. Mitre has assigned CVE-2013-3300 for this bug.
Because the boundary violation only occurs when the third argument is sufficiently large, the output of our automated security scanner was not enough to reveal this data leak. It was only through further manual investigation that we discovered the buffer boundary bug and the security impact it had on our API.
Our takeaways from this experience is outlined as follows:
- Confidence in your code is not enough. Use a solid framework, but make sure it is kept up to date and at least indirectly covered by your test suite. A minor bug in a low level dependency can manifest into a serious security issue.
- Take care when exposing errors across API boundaries. It might be useful to expose internal errors in a development environment, but this should never be done in a production environment.
- Always be testing. Manual security testing is important. Automate as much as you can, but do not simply rely on their output. Go deep into every issue. Custom plugins for tools like OWASP ZAP and Burp Suite can make the difference.
- Check your dependencies for known issues. For Java applications, use OWASP Dependency Check to detect publicly disclosed vulnerabilities in project dependencies. Consider integrating this check into your build tool.
- Developers and security experts should work together. Identifying the symptoms is not enough. It is much easier to identify root causes, evaluate risks, and determine next steps if there is collaboration.
And yes, we’re hiring!
Luca Carettoni, InfoSec
Jerry Zhang, Engineering