« Back to home

EmberData2019

Posted on

For my entry into the #EmberJS2019 blog post compendium this year, I have chosen to focus exclusively on EmberData.

In truth, I have been a bit surprised (and slightly sad) at the shortage of posts that focus on the data story in the Ember ecosystem. I am sure this lack of posts means that folks are completely and highly satisfied with the data story ;-)  If you have your own thoughts after reading this post, please share them! While today (June 17th) marks the end of the call-for-blog-posts, it by no means marks the end of healthy discussion!

I also promise this is not the last of my blog posts about EmberData. I recently tweeted out asking for topic suggestions, and I'll be starting a series based on that soon.

The Complicated Web We Weaved

At #EmberConf2019 this year, Tom and Yehuda presented Octane: our first ever edition. A key consideration in crafting editions is the formulation of a cohesive learning and feature story throughout the ecosystem.

Currently, much of our work in EmberData seems in-cohesive. This year, we would like for our focus in EmberData to be on crafting a cohesive story.

There are four problem areas that I believe EmberData needs to invest in to build this cohesive story, to move EmberData into the modern Ember era, and to be able to provide clear value to 100% of applications.

Problem 1: Class heavy object-oriented-programming model

I often feel that using EmberData feels a lot like writing in an old legacy Ember 1.x codebase: a class heavy object-oriented-programming model - lots of extending, overriding, and Mixin usage - intimate API usage - and strange patterns for working with data that while Javascript feel distinctly un-javascript-y.

It is possible to use EmberData with mostly functional-programming patterns, and indeed doing so usually results in better performance, easier to understand and maintain paradigms, and happier consumers. The library does very little to surface these patterns naturally, though it should.

Problem 2: Overfitting of our modeling layer

Fundamentally, EmberData handles three concerns:

  • Requests: It provides a way to make and manage network requests
  • Cache: It provides a cache for data returned by those requests
  • Presentation: It manages a presentation layer for providing the UI with data in the cache

The biggest value sells of EmberData come primarily from providing application authors with a shared interface through which to access and work with data.  This shared interface serves as a separation of concerns: easing refactoring and enabling routes and components to think less about the specifics of how to fulfill their need for data.  The encapsulation ensures that requests are handled in an orderly manner which allows for authentication and other app-wide concerns to be handled in an easy to find location.

Along the way EmberData accumulated API cruft and became hindered by design patterns that distracted from these core value sells.  We became distracted the most by overfitting the data returned from the API to the classes which present that data to the UI.

This overfitting takes a few forms:

  • We used the UI classes to derive schema information (attribute and relationship definitions). This locked us into requiring a class for every record type that API responses contained. It also has made it more difficult to share that schema information.
  • We encouraged adding adapters and serializers per-type instead of per-API. An Adapter is meant to work with a specific API (a precise spec and format, whether formal or "ad hoc" ). Typically, how you serialize or normalize data depends on the specifics of the API you are connecting to, not the name of the model being presented. Encouraging the per-type pattern leads to applications adding lots of poorly-fit non-reusable adapters and serializers, and the result for many applications is that every new record type added to the API requires a new adapter and serializer to match.

One of the hard lessons that the Javascript ecosystem has struggled with is shipping too much code. Sometimes, when features are needed or specific experiences desired, shipping a lot of bytes over the wire is a good thing. Usually though more bytes is a bad thing. Bytes have a high cost on memory, download and parse times, and to a lesser degree on the runtime (code that is re-used more often is more likely to be optimized, so encouraging reusable patterns can have benefits beyond just the byte size).

EmberData encourages bytes where it doesn't need to. Usually applications have far fewer API versions or ad-hoc API endpoints than model types: encouraging per-API adapters and serializers (or yet an even better paradigm ... stay tuned) would help keep more applications trim as they scale to larger feature sets, more developers, and greater numbers of models.

And what about models themselves? This is where overfitting hurt us the hardest. In truth, EmberData's cache should care less about the specific schema of the data it has, and more about whether or not it has data for a given cache entry. Caring less comes with huge advantages.

  • It fits more use-cases by not locking application author's into the flattened resource structure that EmberData encourages today.
  • It opens the door for a new default-story in which a single model class handles the presentation of any data to the UI. Adding a new type to your API then becomes "cost free".

For caching, the RFCs which introduced RecordData and Identifiers are enabling us to simplify the caching story to be less overfit, although there is more work to be done.

For presentation, the Data team has already done much of the design work needed to enable alternative model classes that will allow us to introduce more scalable designs.

We also have a strong sketch of what designs for schema information and better request primitives should look like.

Completing that design work is one of our top goals for the next 6 months. Doing so will allow us to scale with application size and complexity more naturally, will surface more functional programming paradigms where they make sense to replace some of the object oriented paradigms that fit poorly, and will empower Addon authors to experiment more with adding capability to EmberData.

Problem 3: The library is a Behemoth

As we just discussed, bytes matter. For many Ember applications, EmberData is the largest library within vendor.js. When you use EmberData, you opt into a heavy starting experience. EmberData should provide value for all applications, of any size, that have requirements for fetching, caching, and presenting data.

To achieve this, EmberData needs to lose some weight. Here, fortunately, is a clear roadmap for doing just that as next-steps following the landing of packages.

There's more to this story though. Above I laid out the three concerns EmberData handles to provide value to applications: requesting data, caching data, and presenting data. Each of these three concerns is distinct.

Some applications may only need help managing their requests, without need for a cache or a presentation layer. Others may need request management and caching but not presentation, while still others may need all three. And what exactly the requirements for each of these layers is differs from application to application. In other words: EmberData should be capable of being adopted incrementally and used compositionally.

Much of our design work has focused towards aligning the library for this form of incremental adoption and composition: refining and better defining the interfaces of each. There's more to this story, much more, but it's harder to describe in this post or even in just one post. This is likely a topic I will touch on more either in conference talks or a series of posts at a later time.

Problem 4: Limited Support for Testing

Reminiscent of Ember applications of yonder-years, testing with EmberData feels like the ancient wild west. There is no recommended solution for mocking data, no mechanism for validating that serializers normalize to the expected format, no guides showing how to quickly create records for use in tests, no public mechanisms for creating snapshots for unit and integration style testing of adapters and serialization.

Whether testing your adapter/serializer, some model logic, or generating mock data for a route or component: EmberData leaves it all to you. Thankfully Mirage stepped up to fill some of this gap, but we could do a lot better out of the box.

A cohesive learning story is not just about how you use a library, but the ergonomic wins that come alongside using a standardized solution with standardized tooling. Much as the learning story for components would be incomplete without adequate tooling for testing those components and meaningful errors when using components incorrectly, we want to work towards a best-in-class testing and feedback experience for working with data in your apps.

Separating schema information from presentation classes does more than drive reductions in application size: it provides the infrastructure over which to build a first-class mocking and validation experience.

Problem 5: (bonus) Edge Cases

Before I conclude, I do want to acknowledge that there are many difficult to resolve edge cases users encounter when building with EmberData today. Those edge cases are equally a part of what problems we currently have.

I have strong hopes based on the design work the data team has been doing that most of them have elegant solutions in the near future; however, I feel those are topics for other posts another day. Here I want to stick to the "higher order bits" of the value EmberData provides and the direction the library is headed.

How you could help

I believe the future of Data is bright, but to get there we're going to need more folks to step up to take on advancing the project.

The data team is looking to grow. Whether your strength is in helping us improve our documentation, update patterns and best practices around the eco-system, or land the refactoring and features needed to move EmberData into the modern Ember era, we would love to help you find areas to contribute and grow your contribution.

Attending our weekly meetings (Wednesday's at 2pm Pacific Time), tackling issues labeled "Good for new contributors", or just asking questions in the #dev-ember-data channel on Discord are all great ways to become involved.

We have made great progress towards this bright future, it's time to complete the journey. Finishing what we have started is the most important thing we can do.