Challenges faced while building Neeto commons frontend

At neeto, we are building a lot of products to simplify how we work. Many of these products share similar features such as a 404 page, team member invitations, a sidebar, app switcher, Slack integration, etc. For consistency in both UI and functionality, these common business requirements must remain uniform across all Neeto products.

To bring a new Neeto product to market, we used to copy the whole repo of an already existing product and then we used to delete previous product-specific code from the new repo. This ensured that the visual design, application initialization logic, code quality enforcement rules, etc. are the same in all Neeto products. However, during active development, we noticed three big problems with the consistency of Neeto products.

• Different teams implemented the same business requirements in different ways, causing products to go out of sync.
• Bringing an update to the common functionality required manual changes to every repository.
• Some teams made quick and dirty changes to the common copied logic to fix coding inconveniences.

To address these challenges, we needed a way to share the common code. Simply copying the common code to each repository was not scalable. We built a ruby gem named neeto-commons-backend and an NPM package named neeto-commons-frontend to hold all our common code.

The implementation of the neeto-commons-backend gem was relatively easy, but the implementation of the frontend package, neeto-commons-frontend, posed several challenges. Let's discuss some of the challenges we faced while building neeto-commons-frontend.

public or private?

Both neeto-commons-backend and neeto-commons-frontend contained a lot of business logic specific to neeto. So having these two repos as "private" in Github was an easy call.

When it comes to using neeto-commons-backend gem, we can directly use the gem from GitHub if we configure the access tokens correctly in the host applications. Recently we have deployed a private gem server to host our gems. However, for neeto-commons-frontend, things aren't that straightforward. If we decide to serve the package directly from Github private repository, we will have the following problems:

• Apart from neeto-commons-frontend, we have multiple other frontend packages. To use neeto-commons-frontend as a dependency in them, we will need to hardcode GitHub access token in their package.json file. But, package.json is considered to be a public file. So it is not safe to add any secret keys or tokens to it. If we unknowingly publish any of those packages to npm, our tokens would leak to the public.

• We cannot directly use the ES6 source code in the host application. We need to transpile the JS files before serving. If we were serving neeto-commons-frontend directly from GitHub, we are limited to these options:

  • Add a post_install hook to the package: post_install command is said to be executed at the time of running yarn install or yarn add on the host project. We can add a command to transpile neeto-commons-frontend from that hook. But the post_install hook isn't guaranteed to always run. So it isn't a reliable strategy.
  • Another option is to maintain a copy of transpiled JS output in our GitHub repo by using pre-commit and prepush hooks. But, keeping generated code in version control isn't a good practice. Moreover, can't trust pre-commit and prepush hooks because they can be skipped or can fail to run.

So, we decided to bundle neeto-commons-frontend's JS code using rollup and release it to NPM as a public package. Even though the source code will remain private in GitHub, it would make our bundle available to the public. Anyone can do yarn add @bigbinary/neeto-commons-frontend to obtain our JS bundle.

However, minified JavaScript bundles are nearly impossible to comprehend. Hence we think it's reasonable to make them public. We anyway need the JavaScript bundle to be served publicly in the browsers while loading Neeto products. We cannot keep the frontend JS code completely private.

Release management

In the initial stages of building neeto-commons-frontend, we used to split a large feature into several small sub-issues. So, we raise many small PRs to accomplish a single feature. For this reason, we didn't want to publish a new version of neeto-commons-frontend after merging every PR. We needed manual control over the publishing process.

Also, whenever we do decide to publish a new version we wished to have an automated mechanism to generate release notes explaining the changes from the previous revision.

To satisfy these requirements, we decided to use GitHub releases. It offered the following benefits:

• GitHub can automatically generate release notes using the titles of the merged PRs since the last release.
• The generated release notes will contain a link to visualize code diff between the current and previous releases.
• We can run a GitHub action to automatically publish the package to npm when we create a new release.

We were able to successfully follow that process for a long time. Later on, neeto-commons-frontend became stable. Now, each PR is comprehensive and requires an NPM publish. So, we changed our GitHub action to create a Github release and publish the package to NPM on every PR merge.

common initializers

The neeto-commons-backend gem consists of various code components necessary for initializing the Rails backend, such as configuring CORS, establishing cache, and more.

Likewise, in the frontend, certain initialization tasks must be completed before the React components can begin rendering. These tasks include configuring Axios interceptors and headers, initializing Honeybadger and Mixpanel integrations, setting up translation resources, and more.

Just like neeto-commons-backend, we wanted neeto-commons-frontend to perform all these frontend initialization tasks on its own. But it wasn't as straightforward as we anticipated. Here are some challenges we faced while initializing the host application from neeto-commons-frontend:

Modifying axios instance of the host application

Axios lets us customize its default instance at runtime by adding custom headers. With that, all network requests from our app will have those headers set to it implicitly. Also, Axios lets us register request and response interceptors to view and edit requests and responses before it is sent or received.

All Neeto products use this feature to set Auth token and CSRF token in the headers. They also register interceptors for different use cases like showing toaster messages, handling authorization errors, etc.

Since this logic is the same in all products, we decided to move it to neeto-commons-frontend. Our requirement was to customize the host project's Axios instance from neeto-commons-frontend, without having to write any code from the host project.

For better clarity, let us assume that we are trying to initialize Axios in NeetoCal using neeto-commons-frontend.

To use Axios in neeto-commons-frontend, just like any other JS project, we need to add axios to its package.json. But if we were to add axios as a dependency, rollup will pull out the source code from axios and include it in the package's published bundle.

Similarly, since NeetoCal has both neeto-commons-frontend and axios as its dependencies, webpack will pull out both of their source code and add it to the JS bundle of NeetoCal. It will cause NeetoCal's JS bundle to have two copies of axios code. One from NeetoCal's dependencies and another one from neeto-commons-frontend's bundle.

When a browser loads NeetoCal's bundle, both those codes will get initialized and we will have two separate instances of Axios. Any customizations done from neeto-commons-frontend will be applicable only to its own Axios instance. We won't be able to touch the NeetoCal's Axios instance from neeto-commons-frontend.

As a solution for this, we defined axios as a peerDependency in neeto-commons-frontend's package.json. Since we use rollup-plugin-peer-deps-external plugin, rollup will consider axios as an external dependency while bundling. That means rollup will not pull code from axios and add it to neeto-commons-frontend bundle. Instead, it will keep the import axios from "axios" statement as it is and assume that NeetoCal will have this dependency installed and available at runtime.

Since both NeetoCal and neeto-commons-frontend are now importing Axios from the same source, both of them will share the same instance. Any modifications done from neeto-commons-frontend will reflect on the Axios instance used in the project as well.

Placement of initialization logic

We were unsure of where to initialize the application from. We first tried initializing the app from useEffect hook of the top-most component, App.jsx.

But, as per React's life cycle, the app will run one complete render cycle of all nested components before useEffect gets called. Also, a parent component's useEffect will be executed only after all the useEffects registered in the child components are completed.

This won't work for us due to the following reasons:

• We were using i18next.t() function in several constants to render locale translations. For the translations to be available, we need to initialize i18next before using it. But since constants get initialized immediately after the bundle is loaded, all those calls will result in Translation not found errors.
• Some nested components were performing API calls from their useEffect hooks. Since initialization is not completed by that time, the requests will fail due to missing authentication keys.

To avoid the problem with the delay in useEffect hook, we could have invoked the initialization step directly from the rendering code of App.jsx, by inlining it with the function definition. But, it is not a good practice to introduce side effects from outside useEffect hooks. So, we didn't go with that.

After some trials and errors, we finally decided to place the initialization function call in app/javascript/packs/application.js. It is the file that gets executed before React gets mounted. So our app will be fully initialized before it starts to render.

utility functions

We created neeto-commons-frontend by copying common code from all neeto products to it. We identified these categories of common code: application initialization logic, react components and hooks, and general utility functions.

When copying utility functions, we realized that we could implement a new set of utility functions to minimize boilerplate code in all Neeto products. There are a lot of operations done using array functions like map, filter, find, etc. We were using arrow functions to compare nested properties, which was the most common boilerplate.

We decided to introduce a function matches which checks whether the given pattern is partially equal to the given object. It works like this: the pattern { name: "Oliver" } matches the object { name: "Oliver", phone: 000000 } because the object contains the key name and its value is the same in both the pattern and the object.

With this function as the foundation, we built several array functions like findBy, removeBy, replaceBy, etc. All these functions check for the element that matches the given pattern from an array and performs the required operation on that element.

We also took inspiration from Ramda and implemented currying for such utility functions. This shortened the JS code and made it more declarative.

// before
setUsers(users =>
  users.map(user => (user.address.pincode === 600213 ? newUser : user))
);

// after
setUsers(replaceBy({ address: { pincode: 600213 } }, newUser));

// before
const defaultOrg = organizations.find(({ users }) =>
  users.includes(DEFAULT_USER)
);

// after
const defaultOrg = findBy({ users: includes(DEFAULT_USER) }, organizations);

You can read our blog Extending pure utility functions of Ramda.js to learn more about how we built these utility functions.

dependency management

The utility functions exported by neeto-commons-frontend are like an extension to Ramda. They can be used outside Neeto web applications as well. Several frontend packages and even React Native team can use these utility functions.

Since we import multiple packages as external modules (to use the host project's modules, for example, Axios), we cannot export neeto-commons-frontend as a single bundle. This will force the host applications to have those packages in their dependencies.

To avoid this problem, we decided to create separate bundles for each category. We now have four independent bundles: pure, utils, react-utils, and initializers.

pure bundle contains all the pure functions we have discussed earlier. It needs only Ramda as an external dependency. utils bundle encompasses general utility functions which has dependencies on packages other than Ramda. An example for that is copyToClipboard function. It shows a toaster message if copying is successful. So it depends on @bigbinary/neetoui package as well. react-utils and initializers contains several neeto-specific external dependencies. They are designed to work only on Neeto web apps.

IDE support & types

Initially, all frontend packages at BigBinary were serving UMD bundles. They are compatible with every environment. So there is not much headache of having to publish multiple bundles for different environments.

But UMD bundles do not assist IDEs well. That is, IDE can't provide auto-import, autocompletion, and type support if we distribute UMD packages alone. IDEs even give false positive errors when importing items from the package since they can't detect such an export in the bundle.

The first workaround we tried is to serve ESM or CJS bundles instead of UMD. Both ESM and CJS work well with imports. But imports from the bundle are implicitly typed as any by the IDE. We won't get any predictions for function parameters or component props. We were OK with this setup for a few weeks. This at least does not give false positive errors.

But the problem with this setup is that the developer continuously needs to refer to the docs to understand the parameters a function accepts. This significantly degrades the development experience. To avoid this hassle, some developers preferred not to use our functions and instead wrote lengthy vanilla JS code.

Later we found a solution to this problem. We added explicit type definition using .d.ts files in neeto-commons-frontend package. They contain the type definition of all our exports, written in typescript. We won't copy the JS implementation code to it. It will only contain function declarations.

Since we were exporting four different bundles, we had to add four different .d.ts files with the same name as the bundle. That is, we have pure.d.ts, utils.d.ts, react-utils.d.ts, and initializers.d.ts. The IDE automatically picks up the correct type definition file for the bundle we are importing and uses it to give predictions.

The introduction of type declaration helped us improve the IDE support significantly. It also allows us to add JSDoc comments and deprecation notices for the exported items. Using these, IDE can show documentation for the functions while the developer types.

People not knowing the available functions

Even though we were exporting tons of functions from neeto-commons-frontend, people were not aware of the existence of many of them. So we found that many were reinventing the wheel or wasting their time writing the boilerplate.

As a solution for this, we decided to add some custom ESLint rules to eslint-plugin-neeto which shows warnings to the user about a possible Ramda or neeto-commons-frontend alternative when we detect a corresponding boilerplate code.

You can find the story behind eslint-plugin-neeto and the challenges faced during its development on another blog here.