}
Client: Intel
Data Sense is an ambitious research platform and social network built to help individuals and communities make the most of their personal data.
It’s an incredibly powerful system that combines a general-purpose data-processing backend with a simple-by-design frontend geared toward a casual audience.
Data Sense users can sync their data from 3rd-party cloud services, visualize that data, and create processing pipelines to transform it to perform nearly any kind of analysis. Data Sense’s tightly-integrated social network lets users share personal data (and more) to form a community, with options to contribute while preserving privacy and anonymity.
Our collaboration with Intel Labs made Data Sense simpler and more powerful by streamlining the user experience and frontend architecture.
In the era of privacy leaks and data anxiety, Data Sense set out to understand our relationship to the invisible data clouds we’re all creating.
Some of the issues tackled by the Data Sense research team included:
How can we elevate data literacy among casual users without requiring advanced statistics knowledge?
Do people care about their personal data rights and ownership? Does better access to that data change their minds about that?
Can we create a community centered around sharing personal data while still maintaining privacy and anonymity?
Our contributions to Data Sense were oriented around our design and engineering expertise, and these questions were on our minds with everything we built.
Our primary UX goals for Data Sense were simplicity and unification. To achieve those goals, we worked on several fronts in parallel.
First, we identified the core metaphors used across the site. This was a big challenge because Data Sense exposes a diverse set of abstractions that span multiple complex domains.
Next, we established guidelines for designing simple, consistent UX patterns. Those patterns were chosen for their simplicity, scalability, and generality. They were applied to all new & existing features.
This was a big win because the site was very complex to begin with, and was destined to become even more complex. In addition to being a win for users, simplifying the site at the UX level was a strategic move that let us do more with fewer (human) resources.
Data Sense’s existing features became drastically simpler and more consistent. New features more more tightly-integrated with the rest of the site than they had in the past. And users could (and did) immediately work with greater quantities of much larger datasets.
A key component of Data Sense is its rich community of data experts & enthusiasts. Data Sense users can connect and share their datasets, visualizations, aggregations and insights with each other.
We helped the Data Sense team design and implement this unique social network from the ground up.
First, we created a UX design & engineering strategy to deeply integrate social networking into all new & existing features. This strategy made it possible for users to easily participate in the social network without friction.
We designed and implemented a social-graph-aware search engine to unify access to all Data Sense resources, across all users and feature areas. To address data privacy concerns, we researched ways to integrate Differential Privacy (DP) into Data Sense’s visualizations and its custom data-processing domain specific programming language (DSL).
Building upon our Differential Privacy research, we developed a privacy-preserving social aggregation data visualization. This let users keep tabs on their cohorts while minimizing the risks of data leakage.
In parallel to our UX work, we devised a new software architecture for the Data Sense frontend. This architecture transformed Data Sense from a complex legacy vanilla multi-page app (MPA) based on jQuery and ad-hoc one-off libraries into a simpler single-page app (SPA) based on Vue.JS.
This architecture modularized the frontend, converting hundreds of pages of monolithic scripts into a modular component framework. Shifting interactivity to the frontend simplified the UX even further, by putting more operations on the same page.
Data Sense is a demanding application, serving high volumes of semi-compressible time-series data.
We created a strategy to experimentally find ways to scale Data Sense to thousands of users per server through performance profiling.
First, we wrote automated load-testing scripts deployed on AWS EC2 cloud servers. The load-testing system ran captures of real usage scenarios, incorporating accurate timing & delays. The results of the load tests were automatically processed and rolled up into a report.
After tuning other parts of the system, we identified MySQL server as the primary bottleneck. To find which components of MySQL were causing the bottleneck, we instrumented & profiled MySQL and its supporting infrastructure. This involved finding bottlenecks and testing a matrix of configuration changes. This work followed a rigorous experimental process with end-to-end performance-testing scripts. Every aspect of the system was considered and varied in testing: hardware & network options, MySQL & app server configuration, micro-optimizations, and application architecture.
Data Sense is the unique result of a collaboration between engineers, designers and scientists. It’s an incredibly powerful data-processing platform & social network that has helped thousands unlock insights in their personal data that were previously not possible without learning advanced statistical tools.